2024-10-21 22:37:04,924 [ 125703 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy (runner:42, check_args_and_update_paths) 2024-10-21 22:37:04,924 [ 125703 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration (runner:90, check_args_and_update_paths) 2024-10-21 22:37:04,924 [ 125703 ] INFO : src dir is not set. Will use /home/ubuntu/_work/_temp/test/git-repo-copy/src (runner:97, check_args_and_update_paths) 2024-10-21 22:37:04,924 [ 125703 ] INFO : base_configs_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration (runner:99, check_args_and_update_paths) 2024-10-21 22:37:04,924 [ 125703 ] INFO : Unknown image altinityinfra/dotnet-client (runner:357, ) Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_a8aptb --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_BASE_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_KERBERIZED_HADOOP_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_KERBEROS_KDC_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_MYSQL_JAVA_CLIENT_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_MYSQL_JS_CLIENT_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_MYSQL_PHP_CLIENT_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_NGINX_DAV_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_NEW_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test_allow_list.py::test_allow_list test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_nodes_move/test.py::test_node_move test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_logs_level/test.py::test_check_client_logs_level test_mask_sensitive_info/test.py::test_backup_to_s3 test_mask_sensitive_info/test.py::test_create_alter_user test_mask_sensitive_info/test.py::test_create_database test_mask_sensitive_info/test.py::test_create_dictionary test_mask_sensitive_info/test.py::test_create_table test_mask_sensitive_info/test.py::test_encryption_functions test_mask_sensitive_info/test.py::test_on_cluster test_mask_sensitive_info/test.py::test_table_function_ways_to_call test_mask_sensitive_info/test.py::test_table_functions test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery test_merge_table_over_distributed/test.py::test_filtering test_merge_table_over_distributed/test.py::test_global_in test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' test_part_uuid/test.py::test_part_uuid test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_zookeeper/test.py::test_reload_zookeeper test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replace_partition/test.py::test_normal_work test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_row_policy/test.py::test_dcl_introspection test_row_policy/test.py::test_dcl_management test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml test_row_policy/test.py::test_grant_create_row_policy test_row_policy/test.py::test_introspection test_row_policy/test.py::test_join test_row_policy/test.py::test_miscellaneous_engines test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery test_row_policy/test.py::test_smoke -vvv" altinityinfra/integration-tests-runner:0-46d4f9bd6a40e43bf5543be25cc704364e0c737b '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: repeat-0.9.3, random-0.2, timeout-2.2.0, order-1.0.1, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_keeper_four_word_command/test.py::test_cmd_clrs test_mask_sensitive_info/test.py::test_backup_to_s3 test_reloading_storage_configuration/test.py::test_add_disk test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] test_multi_access_storage_role_management/test.py::test_quotas test_merge_table_over_distributed/test.py::test_filtering test_rocksdb_options/test.py::test_invalid_column_family_options test_log_family_s3/test.py::test_log_family_s3[Log] test_recovery_replica/test.py::test_choose_source_replica [gw8] [ 1%] PASSED test_merge_table_over_distributed/test.py::test_filtering test_merge_table_over_distributed/test.py::test_global_in [gw8] [ 2%] FAILED test_merge_table_over_distributed/test.py::test_global_in test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed [gw8] [ 3%] FAILED test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed [gw0] [ 4%] PASSED test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf [gw0] [ 5%] PASSED test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons [gw0] [ 6%] PASSED test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst [gw0] [ 7%] PASSED test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp [gw0] [ 8%] PASSED test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump [gw0] [ 9%] PASSED test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro [gw0] [ 10%] PASSED test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif [gw0] [ 11%] PASSED test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr [gw6] [ 12%] PASSED test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages [gw0] [ 13%] PASSED test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld [gw7] [ 14%] PASSED test_log_family_s3/test.py::test_log_family_s3[Log] test_log_family_s3/test.py::test_log_family_s3[StripeLog] [gw0] [ 15%] PASSED test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok [gw0] [ 16%] PASSED test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst [gw0] [ 17%] PASSED test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr [gw0] [ 18%] PASSED test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_session/test.py::test_session_close_shutdown [gw0] [ 19%] PASSED test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc [gw0] [ 20%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp [gw7] [ 21%] PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] test_log_family_s3/test.py::test_log_family_s3[TinyLog] [gw0] [ 22%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs [gw0] [ 23%] PASSED test_keeper_four_word_command/test.py::test_cmd_wchs [gw5] [ 24%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] [gw7] [ 25%] PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] [gw4] [ 26%] PASSED test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_keeper_force_recovery_single_node/test.py::test_cluster_recovery [gw1] [ 27%] PASSED test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with [gw5] [ 28%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] [gw2] [ 29%] PASSED test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy [gw6] [ 30%] PASSED test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles test_row_policy/test.py::test_change_of_users_xml_changes_row_policies [gw5] [ 31%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] [gw5] [ 32%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] [gw6] [ 33%] PASSED test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles [gw5] [ 34%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] [gw2] [ 35%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config [gw4] [ 36%] PASSED test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_column_family_options [gw9] [ 37%] PASSED test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery [gw5] [ 38%] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] [gw6] [ 39%] PASSED test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users [gw2] [ 40%] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy [gw9] [ 41%] PASSED test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata [gw6] [ 42%] PASSED test_multi_access_storage_role_management/test.py::test_users test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit [gw1] [ 43%] FAILED test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_keeper_nodes_move/test.py::test_node_move [gw1] [ 43%] ERROR test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_row_policy/test.py::test_dcl_introspection [gw4] [ 44%] PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task [gw2] [ 45%] PASSED test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy [gw8] [ 46%] PASSED test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout [gw8] [ 47%] PASSED test_keeper_session/test.py::test_session_timeout test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch [gw2] [ 48%] PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works [gw1] [ 49%] FAILED test_row_policy/test.py::test_dcl_introspection [gw1] [ 49%] ERROR test_row_policy/test.py::test_dcl_introspection test_row_policy/test.py::test_dcl_management [gw1] [ 50%] FAILED test_row_policy/test.py::test_dcl_management [gw1] [ 50%] ERROR test_row_policy/test.py::test_dcl_management test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml [gw4] [ 51%] PASSED test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options [gw4] [ 52%] PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options [gw4] [ 53%] PASSED test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options [gw4] [ 54%] PASSED test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options [gw4] [ 55%] PASSED test_rocksdb_options/test.py::test_valid_options [gw2] [ 56%] PASSED test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings [gw9] [ 57%] PASSED test_recovery_replica/test.py::test_update_metadata [gw1] [ 58%] FAILED test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml [gw1] [ 58%] ERROR test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml test_row_policy/test.py::test_grant_create_row_policy [gw1] [ 59%] FAILED test_row_policy/test.py::test_grant_create_row_policy [gw1] [ 59%] ERROR test_row_policy/test.py::test_grant_create_row_policy test_row_policy/test.py::test_introspection [gw2] [ 60%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery [gw3] [ 61%] PASSED test_mask_sensitive_info/test.py::test_backup_to_s3 test_mask_sensitive_info/test.py::test_create_alter_user [gw0] [ 62%] PASSED test_keeper_force_recovery_single_node/test.py::test_cluster_recovery [gw2] [ 63%] PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy [gw3] [ 64%] PASSED test_mask_sensitive_info/test.py::test_create_alter_user test_mask_sensitive_info/test.py::test_create_database [gw4] [ 65%] PASSED test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings test_keeper_four_word_command/test_allow_list.py::test_allow_list [gw1] [ 66%] FAILED test_row_policy/test.py::test_introspection [gw1] [ 66%] ERROR test_row_policy/test.py::test_introspection test_row_policy/test.py::test_join [gw3] [ 67%] PASSED test_mask_sensitive_info/test.py::test_create_database test_mask_sensitive_info/test.py::test_create_dictionary [gw8] [ 68%] PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns [gw8] [ 69%] PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns [gw1] [ 70%] FAILED test_row_policy/test.py::test_join [gw3] [ 71%] PASSED test_mask_sensitive_info/test.py::test_create_dictionary test_mask_sensitive_info/test.py::test_create_table [gw1] [ 71%] ERROR test_row_policy/test.py::test_join test_row_policy/test.py::test_miscellaneous_engines [gw2] [ 72%] PASSED test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_replication_without_zookeeper/test.py::test_startup_without_zookeeper [gw1] [ 73%] FAILED test_row_policy/test.py::test_miscellaneous_engines [gw1] [ 73%] ERROR test_row_policy/test.py::test_miscellaneous_engines test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned [gw6] [ 74%] PASSED test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit test_part_uuid/test.py::test_part_uuid [gw1] [ 75%] FAILED test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned [gw1] [ 75%] ERROR test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role [gw2] [ 76%] PASSED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk [gw0] [ 77%] PASSED test_keeper_four_word_command/test_allow_list.py::test_allow_list [gw1] [ 78%] FAILED test_row_policy/test.py::test_policy_on_distributed_table_via_role [gw1] [ 78%] ERROR test_row_policy/test.py::test_policy_on_distributed_table_via_role test_row_policy/test.py::test_reload_users_xml_by_timer [gw1] [ 79%] FAILED test_row_policy/test.py::test_reload_users_xml_by_timer [gw1] [ 79%] ERROR test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery [gw9] [ 80%] PASSED test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery [gw1] [ 81%] FAILED test_row_policy/test.py::test_row_policy_filter_with_subquery [gw1] [ 81%] ERROR test_row_policy/test.py::test_row_policy_filter_with_subquery test_row_policy/test.py::test_smoke test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster [gw1] [ 82%] FAILED test_row_policy/test.py::test_smoke [gw2] [ 83%] PASSED test_reloading_storage_configuration/test.py::test_update_disk [gw3] [ 84%] PASSED test_mask_sensitive_info/test.py::test_create_table test_mask_sensitive_info/test.py::test_encryption_functions [gw1] [ 84%] ERROR test_row_policy/test.py::test_smoke [gw3] [ 85%] FAILED test_mask_sensitive_info/test.py::test_encryption_functions test_mask_sensitive_info/test.py::test_on_cluster test_replace_partition/test.py::test_normal_work [gw3] [ 86%] PASSED test_mask_sensitive_info/test.py::test_on_cluster test_mask_sensitive_info/test.py::test_table_function_ways_to_call [gw5] [ 87%] PASSED test_keeper_nodes_move/test.py::test_node_move test_logs_level/test.py::test_check_client_logs_level [gw6] [ 88%] PASSED test_part_uuid/test.py::test_part_uuid [gw9] [ 89%] FAILED test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions [gw5] [ 90%] PASSED test_logs_level/test.py::test_check_client_logs_level [gw4] [ 91%] PASSED test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop [gw6] [ 92%] PASSED test_read_temporary_tables_on_failure/test.py::test_different_versions [gw3] [ 93%] PASSED test_mask_sensitive_info/test.py::test_table_function_ways_to_call test_mask_sensitive_info/test.py::test_table_functions [gw5] [ 94%] PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop [gw0] [ 95%] PASSED test_keeper_mntr_pressure/test.py::test_aggressive_mntr [gw8] [ 96%] PASSED test_replace_partition/test.py::test_normal_work [gw3] [ 97%] PASSED test_mask_sensitive_info/test.py::test_table_functions [gw7] [ 98%] FAILED test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table [gw7] [ 99%] PASSED test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table test_reload_zookeeper/test.py::test_reload_zookeeper [gw7] [100%] PASSED test_reload_zookeeper/test.py::test_reload_zookeeper ==================================== ERRORS ==================================== ______ ERROR at teardown of test_change_of_users_xml_changes_row_policies ______ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:37:39 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:37:39 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:37:39 [ 373 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node (cluster.py:3423, query) 2024-10-21 22:37:39 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node2_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:37:39 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node2_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:37:39 [ 373 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node2 (cluster.py:3423, query) 2024-10-21 22:37:39 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) 2024-10-21 22:37:39 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table2 on node (cluster.py:3423, query) 2024-10-21 22:37:40 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table3 on node (cluster.py:3423, query) 2024-10-21 22:37:40 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:37:40 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:37:40 [ 373 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node (cluster.py:3423, query) 2024-10-21 22:37:40 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node2_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:37:40 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node2_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:37:40 [ 373 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node2 (cluster.py:3423, query) 2024-10-21 22:37:40 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:37:56 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:37:56 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:37:56 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:37:56 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _________________ ERROR at teardown of test_dcl_introspection __________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:37:57 [ 373 ] DEBUG : Executing query SHOW POLICIES on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:07 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:07 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:07 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:07 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ___________________ ERROR at teardown of test_dcl_management ___________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:08 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:08 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:08 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:08 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:08 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:08 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:08 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:08 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _______ ERROR at teardown of test_dcl_users_with_policies_from_users_xml _______ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:08 [ 373 ] DEBUG : Executing query CREATE USER X on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:19 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:19 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:19 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:19 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ______________ ERROR at teardown of test_grant_create_row_policy _______________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:19 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:19 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:19 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:19 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:19 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:19 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:19 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:19 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ___________________ ERROR at teardown of test_introspection ____________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:19 [ 373 ] DEBUG : Executing query SELECT * from system.row_policies ORDER BY short_name, database, table on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:30 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:30 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:30 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:30 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ________________________ ERROR at teardown of test_join ________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:30 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 as t1 ANY LEFT JOIN mydb.filtered_table1 as t2 ON t1.a = t2.b on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:34 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:34 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:34 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:34 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _______________ ERROR at teardown of test_miscellaneous_engines ________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:34 [ 373 ] DEBUG : Executing query CREATE ROW POLICY OR REPLACE pC ON mydb.other_table FOR SELECT USING a = 1 TO default on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:37 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:37 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:37 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:37 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) __ ERROR at teardown of test_policy_from_users_xml_affects_only_user_assigned __ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:37 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:40 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:40 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:40 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:40 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ________ ERROR at teardown of test_policy_on_distributed_table_via_role ________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:40 [ 373 ] DEBUG : Executing query DROP TABLE IF EXISTS local_tbl on node (cluster.py:3423, query) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:43 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:43 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _____________ ERROR at teardown of test_reload_users_xml_by_timer ______________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:43 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:43 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:43 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:43 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) __________ ERROR at teardown of test_row_policy_filter_with_subquery ___________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:43 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:43 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:44 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:44 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:44 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:44 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _______________________ ERROR at teardown of test_smoke ________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(autouse=True) def reset_policies(): try: yield finally: > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:87: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:44 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) --------------------------- Captured stderr teardown --------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:38:46 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:46 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:46 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:46 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) 2024-10-21 22:38:46 [ 373 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/.env', '--project-name', 'roottestrowpolicy', '--file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/node/docker-compose.yml', '--file', '/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/node2/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Stderr:Stopping roottestrowpolicy_node2_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo1_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo2_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo3_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Stderr:Stopping roottestrowpolicy_node2_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo2_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo1_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Stderr:Stopping roottestrowpolicy_zoo3_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_row_policy/_instances_0/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_row_policy/_instances_0/node/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_row_policy/_instances_0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_row_policy/_instances_0/node2/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-10-21 22:38:51 [ 373 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/.env', '--project-name', 'roottestrowpolicy', '--file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/node/docker-compose.yml', '--file', '/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_row_policy/_instances_0/node2/docker-compose.yml', 'down', '--volumes'] (cluster.py:105, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing roottestrowpolicy_node_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing roottestrowpolicy_node2_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo1_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo2_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo3_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo1_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing roottestrowpolicy_node2_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo3_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing roottestrowpolicy_zoo2_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing roottestrowpolicy_node_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stderr:Removing network roottestrowpolicy_default (cluster.py:115, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-10-21 22:38:52 [ 373 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 26 Running: 21 Paused: 0 Stopped: 5 Images: 41 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: 2110fc98d269 ID: f2e62ed7-a1b0-4a54-9efb-9f1d0b410eca Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-10-21 22:38:52 [ 373 ] DEBUG : Docker networks for project roottestrowpolicy are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-10-21 22:38:52 [ 373 ] DEBUG : Docker containers for project roottestrowpolicy are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-10-21 22:38:52 [ 373 ] DEBUG : Docker volumes for project roottestrowpolicy are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-10-21 22:38:52 [ 373 ] DEBUG : Command:docker container list --all --filter name='^/roottestrowpolicy_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-10-21 22:38:52 [ 373 ] DEBUG : No running containers for project: roottestrowpolicy (cluster.py:782, cleanup) 2024-10-21 22:38:52 [ 373 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-10-21 22:38:52 [ 373 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-10-21 22:38:52 [ 373 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-10-21 22:38:52 [ 373 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-10-21 22:38:52 [ 373 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-10-21 22:38:52 [ 373 ] DEBUG : Stdout:3 (cluster.py:113, run_and_check) =================================== FAILURES =================================== ________________________________ test_global_in ________________________________ [gw8] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_global_in(started_cluster): assert ( node1.query( "SELECT val FROM distributed_table WHERE id GLOBAL IN (SELECT toUInt32(3 - id) FROM local_table)" ).rstrip() == "node2" ) > assert ( node1.query( "SELECT val FROM merge_table WHERE id GLOBAL IN (SELECT toUInt32(3 - id) FROM local_table)" ).rstrip() == "node2" ) test_merge_table_over_distributed/test.py:50: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 47, stderr: Received exception from server (version 23.8.16): E Code: 47. DB::Exception: Received from 172.16.2.3:9000. DB::Exception: Unknown expression or table expression identifier 'default._data_17601879111361243894_17988131190591442068' in scope SELECT local_table.val AS val FROM default.local_table WHERE local_table.id GLOBAL IN (default._data_17601879111361243894_17988131190591442068). Stack trace: E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e082673 in /usr/bin/clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5cce50 in /usr/bin/clickhouse E 2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type, std::type_identity::type, std::type_identity::type>, char const*&&, String&, String const&, String&&, String&&) @ 0x0000000031302de7 in /usr/bin/clickhouse E 3. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5484: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x0000000031239357 in /usr/bin/clickhouse E 4. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:5688: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNodeList(std::shared_ptr&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x0000000031233747 in /usr/bin/clickhouse E 5. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveFunction(std::shared_ptr&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000003126cab2 in /usr/bin/clickhouse E 6. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::(anonymous namespace)::QueryAnalyzer::resolveExpressionNode(std::shared_ptr&, DB::(anonymous namespace)::IdentifierResolveScope&, bool, bool) @ 0x0000000031235a2e in /usr/bin/clickhouse E 7. ./build_docker/./contrib/llvm-project/libcxx/include/vector:580: DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000003122097c in /usr/bin/clickhouse E 8. ./build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:0: DB::QueryAnalysisPass::run(std::shared_ptr, std::shared_ptr) @ 0x000000003121a876 in /usr/bin/clickhouse E 9. ./build_docker/./src/Analyzer/QueryTreePassManager.cpp:0: DB::QueryTreePassManager::run(std::shared_ptr, unsigned long) @ 0x0000000031216128 in /usr/bin/clickhouse E 10. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr const&, DB::SelectQueryOptions const&, std::shared_ptr const&, std::shared_ptr const&) @ 0x00000000315fce97 in /usr/bin/clickhouse E 11. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:161: DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr const&, std::shared_ptr const&, DB::SelectQueryOptions const&) @ 0x00000000315f9789 in /usr/bin/clickhouse E 12. ./build_docker/./src/Processors/QueryPlan/DistributedCreateLocalPlan.cpp:92: DB::createLocalPlan(std::shared_ptr const&, DB::Block const&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long, unsigned long, unsigned long, std::shared_ptr, StrongTypedef, DB::UUIDTag>) @ 0x0000000035c0d18b in /usr/bin/clickhouse E 13. ./build_docker/./src/Interpreters/ClusterProxy/SelectStreamFactory.cpp:126: DB::ClusterProxy::SelectStreamFactory::createForShard(DB::Cluster::ShardInfo const&, std::shared_ptr const&, DB::StorageID const&, std::shared_ptr const&, std::shared_ptr, std::vector>, std::allocator>>>&, std::vector>&, unsigned int, bool)::$_0::operator()() const @ 0x0000000032e2759a in /usr/bin/clickhouse E 14. ./build_docker/./src/Interpreters/ClusterProxy/SelectStreamFactory.cpp:190: DB::ClusterProxy::SelectStreamFactory::createForShard(DB::Cluster::ShardInfo const&, std::shared_ptr const&, DB::StorageID const&, std::shared_ptr const&, std::shared_ptr, std::vector>, std::allocator>>>&, std::vector>&, unsigned int, bool) @ 0x0000000032e2481c in /usr/bin/clickhouse E 15. ./build_docker/./src/Interpreters/ClusterProxy/executeQuery.cpp:220: DB::ClusterProxy::executeQuery(DB::QueryPlan&, DB::Block const&, DB::QueryProcessingStage::Enum, DB::StorageID const&, std::shared_ptr const&, DB::ClusterProxy::SelectStreamFactory&, Poco::Logger*, std::shared_ptr const&, std::shared_ptr, DB::SelectQueryInfo const&, std::shared_ptr const&, String const&, std::shared_ptr const&, std::function (unsigned long)>) @ 0x0000000032e30d42 in /usr/bin/clickhouse E 16. ./build_docker/./src/Storages/StorageDistributed.cpp:802: DB::StorageDistributed::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x0000000032de24ba in /usr/bin/clickhouse E 17. ./build_docker/./src/Storages/StorageMerge.cpp:726: DB::ReadFromMerge::createSources(DB::QueryProcessingStage::Enum, unsigned long, DB::Block const&, std::tuple, std::shared_ptr, String> const&, std::vector>, DB::Block const&, std::shared_ptr, unsigned long, bool) @ 0x0000000033056851 in /usr/bin/clickhouse E 18. ./build_docker/./src/Storages/StorageMerge.cpp:496: DB::ReadFromMerge::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000033052734 in /usr/bin/clickhouse E 19. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector>, std::allocator>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c495d2 in /usr/bin/clickhouse E 20. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000035c9ad33 in /usr/bin/clickhouse E 21. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:252: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000315ffe18 in /usr/bin/clickhouse E 22. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:436: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000315ff607 in /usr/bin/clickhouse E 23. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320eecdd in /usr/bin/clickhouse E 24. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320e5aa1 in /usr/bin/clickhouse E 25. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ed6d3d in /usr/bin/clickhouse E 26. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f13b30 in /usr/bin/clickhouse E 27. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcb69cf in /usr/bin/clickhouse E 28. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcb7797 in /usr/bin/clickhouse E 29. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e196fbc in /usr/bin/clickhouse E 30. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e190068 in /usr/bin/clickhouse E 31. ? @ 0x00007f8a405c9ac3 in ? E . (UNKNOWN_IDENTIFIER) E (query: SELECT val FROM merge_table WHERE id GLOBAL IN (SELECT toUInt32(3 - id) FROM local_table)) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:37:26 [ 394 ] DEBUG : Executing query SELECT val FROM distributed_table WHERE id GLOBAL IN (SELECT toUInt32(3 - id) FROM local_table) on node1 (cluster.py:3423, query) 2024-10-21 22:37:26 [ 394 ] DEBUG : Executing query SELECT val FROM merge_table WHERE id GLOBAL IN (SELECT toUInt32(3 - id) FROM local_table) on node1 (cluster.py:3423, query) ______________ test_select_table_name_from_merge_over_distributed ______________ [gw8] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_select_table_name_from_merge_over_distributed(started_cluster): node1.query("INSERT INTO local_table_2 VALUES (1, 'node1')") node2.query("INSERT INTO local_table_2 VALUES (2, 'node2')") > node1.query("select _table == 'distributed_table' from merge_table") test_merge_table_over_distributed/test.py:79: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 10, stderr: Received exception from server (version 23.8.16): E Code: 10. DB::Exception: Received from 172.16.2.3:9000. DB::Exception: Column `_table` not found in table default.merge_table (09bafc76-b80c-4010-8549-5a13ab6cba77). Stack trace: E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e082673 in /usr/bin/clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5cce50 in /usr/bin/clickhouse E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type, std::type_identity::type>, String&&, String&&) @ 0x000000000bd89e96 in /usr/bin/clickhouse E 3. ./build_docker/./src/Storages/StorageSnapshot.cpp:142: DB::StorageSnapshot::getSampleBlockForColumns(std::vector> const&) const @ 0x000000003364ca38 in /usr/bin/clickhouse E 4. ./build_docker/./src/Planner/PlannerJoinTree.cpp:804: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x0000000031667428 in /usr/bin/clickhouse E 5. ./build_docker/./src/Planner/PlannerJoinTree.cpp:1450: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000003165712a in /usr/bin/clickhouse E 6. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x0000000031608256 in /usr/bin/clickhouse E 7. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000316025c7 in /usr/bin/clickhouse E 8. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getSampleBlock() @ 0x00000000315ff1d6 in /usr/bin/clickhouse E 9. ./build_docker/./src/Interpreters/getHeaderForProcessingStage.cpp:0: DB::getHeaderForProcessingStage(std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo const&, std::shared_ptr, DB::QueryProcessingStage::Enum) @ 0x0000000032121df8 in /usr/bin/clickhouse E 10. ./build_docker/./src/Storages/StorageMerge.cpp:358: DB::StorageMerge::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x000000003304eef3 in /usr/bin/clickhouse E 11. ./build_docker/./src/Planner/PlannerJoinTree.cpp:756: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x0000000031666617 in /usr/bin/clickhouse E 12. ./build_docker/./src/Planner/PlannerJoinTree.cpp:1450: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000003165712a in /usr/bin/clickhouse E 13. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x0000000031608256 in /usr/bin/clickhouse E 14. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000316025c7 in /usr/bin/clickhouse E 15. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x0000000031600212 in /usr/bin/clickhouse E 16. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320ede1d in /usr/bin/clickhouse E 17. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320e5aa1 in /usr/bin/clickhouse E 18. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ed6d3d in /usr/bin/clickhouse E 19. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f13b30 in /usr/bin/clickhouse E 20. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcb69cf in /usr/bin/clickhouse E 21. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcb7797 in /usr/bin/clickhouse E 22. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e196fbc in /usr/bin/clickhouse E 23. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e190068 in /usr/bin/clickhouse E 24. ? @ 0x00007f8a405c9ac3 in ? E 25. ? @ 0x00007f8a4065b850 in ? E . (NOT_FOUND_COLUMN_IN_BLOCK) E (query: select _table == 'distributed_table' from merge_table) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:37:26 [ 394 ] DEBUG : Executing query INSERT INTO local_table_2 VALUES (1, 'node1') on node1 (cluster.py:3423, query) 2024-10-21 22:37:26 [ 394 ] DEBUG : Executing query INSERT INTO local_table_2 VALUES (2, 'node2') on node2 (cluster.py:3423, query) 2024-10-21 22:37:27 [ 394 ] DEBUG : Executing query select _table == 'distributed_table' from merge_table on node1 (cluster.py:3423, query) --------------------------- Captured stderr teardown --------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:37:27 [ 394 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/.env', '--project-name', 'roottestmergetableoverdistributed', '--file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node2/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check) 2024-10-21 22:37:30 [ 394 ] DEBUG : Stderr:Stopping roottestmergetableoverdistributed_node1_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:37:30 [ 394 ] DEBUG : Stderr:Stopping roottestmergetableoverdistributed_node2_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:37:30 [ 394 ] DEBUG : Stderr:Stopping roottestmergetableoverdistributed_node1_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:37:30 [ 394 ] DEBUG : Stderr:Stopping roottestmergetableoverdistributed_node2_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:37:30 [ 394 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node1/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-10-21 22:37:30 [ 394 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node2/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-10-21 22:37:31 [ 394 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/.env', '--project-name', 'roottestmergetableoverdistributed', '--file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_merge_table_over_distributed/_instances_0/node2/docker-compose.yml', 'down', '--volumes'] (cluster.py:105, run_and_check) 2024-10-21 22:37:31 [ 394 ] DEBUG : Stderr:Removing roottestmergetableoverdistributed_node1_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:37:31 [ 394 ] DEBUG : Stderr:Removing roottestmergetableoverdistributed_node2_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:37:31 [ 394 ] DEBUG : Stderr:Removing roottestmergetableoverdistributed_node2_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:37:31 [ 394 ] DEBUG : Stderr:Removing roottestmergetableoverdistributed_node1_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:37:31 [ 394 ] DEBUG : Stderr:Removing network roottestmergetableoverdistributed_default (cluster.py:115, run_and_check) 2024-10-21 22:37:31 [ 394 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-10-21 22:37:31 [ 394 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 29 Running: 29 Paused: 0 Stopped: 0 Images: 41 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: 2110fc98d269 ID: f2e62ed7-a1b0-4a54-9efb-9f1d0b410eca Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-10-21 22:37:31 [ 394 ] DEBUG : Docker networks for project roottestmergetableoverdistributed are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-10-21 22:37:31 [ 394 ] DEBUG : Docker containers for project roottestmergetableoverdistributed are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-10-21 22:37:31 [ 394 ] DEBUG : Docker volumes for project roottestmergetableoverdistributed are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-10-21 22:37:31 [ 394 ] DEBUG : Command:docker container list --all --filter name='^/roottestmergetableoverdistributed_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-10-21 22:37:31 [ 394 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-10-21 22:37:31 [ 394 ] DEBUG : No running containers for project: roottestmergetableoverdistributed (cluster.py:782, cleanup) 2024-10-21 22:37:31 [ 394 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-10-21 22:37:31 [ 394 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-10-21 22:37:31 [ 394 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-10-21 22:37:31 [ 394 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-10-21 22:37:31 [ 394 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-10-21 22:37:31 [ 394 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-10-21 22:37:31 [ 394 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-10-21 22:37:32 [ 394 ] DEBUG : Stdout:3 (cluster.py:113, run_and_check) ________________ test_change_of_users_xml_changes_row_policies _________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_change_of_users_xml_changes_row_policies(): copy_policy_xml("normal_filters.xml") assert node.query("SELECT * FROM mydb.filtered_table1") == TSV([[1, 0], [1, 1]]) assert node.query("SELECT * FROM mydb.filtered_table2") == TSV( [[0, 0, 0, 0], [0, 0, 6, 0]] ) assert node.query("SELECT * FROM mydb.filtered_table3") == TSV([[0, 1], [1, 0]]) copy_policy_xml("all_rows.xml") > assert node.query("SELECT * FROM mydb.filtered_table1") == TSV( [[0, 0], [0, 1], [1, 0], [1, 1]] ) test_row_policy/test.py:328: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 32, stderr: [node] 2024.10.21 22:37:41.022844 [ 300 ] BaseDaemon: ######################################## E [node] 2024.10.21 22:37:41.022964 [ 300 ] BaseDaemon: (version 23.8.16.42.altinitystable (altinity build), build id: 411D8AE124013E7DADB5519CC590EE3B4C3521B5, git hash: 46d4f9bd6a40e43bf5543be25cc704364e0c737b) (from thread 8) (query_id: 1a35eaff-85e5-4cb1-8cc4-14eeedcdbe4d) (query: SELECT * FROM mydb.filtered_table1) Received signal Segmentation fault (11) E [node] 2024.10.21 22:37:41.023023 [ 300 ] BaseDaemon: Address: NULL pointer. Access: read. Address not mapped to object. E [node] 2024.10.21 22:37:41.023070 [ 300 ] BaseDaemon: Stack trace: 0x0000557102114a4d 0x00005571028ee035 0x00007f629d473520 0x0000557115c5e01c 0x00005571160ee6a7 0x000055711611236e 0x000055711610612a 0x00005571160b7256 0x00005571160b15c7 0x00005571160af212 0x0000557116b9ce1d 0x0000557116b94aa1 0x0000557119985d3d 0x00005571199c2b30 0x00005571227659cf 0x0000557122766797 0x0000557122c45fbc 0x0000557122c3f068 0x00007f629d4c5ac3 0x00007f629d557850 E [node] 2024.10.21 22:37:41.105480 [ 300 ] BaseDaemon: 0.1. inlined from ./build_docker/./src/Common/StackTrace.cpp:286: StackTrace::tryCapture() E [node] 2024.10.21 22:37:41.105586 [ 300 ] BaseDaemon: 0. ./build_docker/./src/Common/StackTrace.cpp:258: StackTrace::StackTrace(ucontext_t const&) @ 0x000000001d665a4d in /usr/bin/clickhouse E [node] 2024.10.21 22:37:41.361862 [ 300 ] BaseDaemon: 1.1. inlined from ./build_docker/./src/IO/WriteHelpers.h:88: void DB::writePODBinary(int const&, DB::WriteBuffer&) E [node] 2024.10.21 22:37:41.361959 [ 300 ] BaseDaemon: 1.2. inlined from ./build_docker/./src/IO/WriteHelpers.h:881: void DB::writeBinary(int const&, DB::WriteBuffer&) E [node] 2024.10.21 22:37:41.362017 [ 300 ] BaseDaemon: 1. ./build_docker/./src/Daemon/BaseDaemon.cpp:152: signalHandler(int, siginfo_t*, void*) @ 0x000000001de3f035 in /usr/bin/clickhouse E [node] 2024.10.21 22:37:41.362066 [ 300 ] BaseDaemon: 2. ? @ 0x00007f629d473520 in ? E [node] 2024.10.21 22:37:41.703522 [ 300 ] BaseDaemon: 3.1. inlined from ./build_docker/./src/Analyzer/QueryTreeBuilder.cpp:123: QueryTreeBuilder E [node] 2024.10.21 22:37:41.703658 [ 300 ] BaseDaemon: 3. ./build_docker/./src/Analyzer/QueryTreeBuilder.cpp:1029: DB::buildQueryTree(std::shared_ptr, std::shared_ptr) @ 0x00000000311af01c in /usr/bin/clickhouse E [node] 2024.10.21 22:37:42.042549 [ 300 ] BaseDaemon: 4. ./build_docker/./src/Planner/Utils.cpp:0: DB::buildFilterInfo(std::shared_ptr, std::shared_ptr const&, std::shared_ptr&, std::unordered_set, std::equal_to, std::allocator>) @ 0x000000003163f6a7 in /usr/bin/clickhouse E [node] 2024.10.21 22:37:42.581855 [ 300 ] BaseDaemon: 5.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__hash_table:1473: ~__hash_table E [node] 2024.10.21 22:37:42.581938 [ 300 ] BaseDaemon: 5.2. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/unordered_set:615: ~unordered_set E [node] 2024.10.21 22:37:42.581989 [ 300 ] BaseDaemon: 5.3. inlined from ./build_docker/./src/Planner/PlannerJoinTree.cpp:447: DB::(anonymous namespace)::buildRowPolicyFilterIfNeeded(std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr&) E [node] 2024.10.21 22:37:42.582026 [ 300 ] BaseDaemon: 5. ./build_docker/./src/Planner/PlannerJoinTree.cpp:730: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x000000003166336e in /usr/bin/clickhouse E [node] 2024.10.21 22:37:43.142923 [ 300 ] BaseDaemon: 6. ./build_docker/./src/Planner/PlannerJoinTree.cpp:1450: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000003165712a in /usr/bin/clickhouse E [node] 2024.10.21 22:37:43.485859 [ 300 ] BaseDaemon: 7. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x0000000031608256 in /usr/bin/clickhouse E [node] 2024.10.21 22:37:43.790045 [ 300 ] BaseDaemon: 8. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000316025c7 in /usr/bin/clickhouse E [node] 2024.10.21 22:37:43.929598 [ 300 ] BaseDaemon: 9.1. inlined from ./build_docker/./src/Planner/Planner.h:44: DB::Planner::getQueryPlan() E [node] 2024.10.21 22:37:43.929697 [ 300 ] BaseDaemon: 9. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:235: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x0000000031600212 in /usr/bin/clickhouse E [node] 2024.10.21 22:37:44.395578 [ 300 ] BaseDaemon: 10. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320ede1d in /usr/bin/clickhouse E [node] 2024.10.21 22:37:44.932174 [ 300 ] BaseDaemon: 11. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320e5aa1 in /usr/bin/clickhouse E [node] 2024.10.21 22:37:45.208307 [ 300 ] BaseDaemon: 12. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ed6d3d in /usr/bin/clickhouse E [node] 2024.10.21 22:37:45.756282 [ 300 ] BaseDaemon: 13. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f13b30 in /usr/bin/clickhouse E [node] 2024.10.21 22:37:45.766272 [ 300 ] BaseDaemon: 14. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcb69cf in /usr/bin/clickhouse E [node] 2024.10.21 22:37:45.786556 [ 300 ] BaseDaemon: 15.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: std::default_delete::operator()[abi:v15000](Poco::Net::TCPServerConnection*) const E [node] 2024.10.21 22:37:45.786646 [ 300 ] BaseDaemon: 15.2. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:305: std::unique_ptr>::reset[abi:v15000](Poco::Net::TCPServerConnection*) E [node] 2024.10.21 22:37:45.786703 [ 300 ] BaseDaemon: 15.3. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259: ~unique_ptr E [node] 2024.10.21 22:37:45.786744 [ 300 ] BaseDaemon: 15. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:116: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcb7797 in /usr/bin/clickhouse E [node] 2024.10.21 22:37:45.808918 [ 300 ] BaseDaemon: 16. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e196fbc in /usr/bin/clickhouse E [node] 2024.10.21 22:37:45.831837 [ 300 ] BaseDaemon: 17. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e190068 in /usr/bin/clickhouse E [node] 2024.10.21 22:37:45.831917 [ 300 ] BaseDaemon: 18. ? @ 0x00007f629d4c5ac3 in ? E [node] 2024.10.21 22:37:45.831948 [ 300 ] BaseDaemon: 19. ? @ 0x00007f629d557850 in ? E [node] 2024.10.21 22:37:47.362115 [ 300 ] BaseDaemon: Integrity check of the executable successfully passed (checksum: 107BDC2E6B5B34F49E6DFC9872580706) E [node] 2024.10.21 22:37:52.219964 [ 300 ] BaseDaemon: You are using an Altinity Stable Build. Please log issues at https://github.com/Altinity/ClickHouse/issues. Thank you! E [node] 2024.10.21 22:37:52.220176 [ 300 ] BaseDaemon: Changed settings: any_join_distinct_right_table_keys = true, allow_experimental_analyzer = true, optimize_on_insert = false E Error on processing query: Code: 32. DB::Exception: Attempt to read after eof: while receiving packet from 172.16.9.5:9000. (ATTEMPT_TO_READ_AFTER_EOF), Stack trace (when copying this message, always include the lines below): E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e082673 in /clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5cce50 in /clickhouse E 2. DB::Exception::Exception(int, char const (&) [26]) @ 0x000000000c5ee509 in /clickhouse E 3. ./build_docker/./src/IO/VarInt.cpp:13: DB::throwReadAfterEOF() @ 0x000000001d75d07e in /clickhouse E 4. ./build_docker/./src/QueryPipeline/ProfileInfo.h:18: DB::Connection::receivePacket() @ 0x0000000034c18be3 in /clickhouse E 5. ./build_docker/./src/Client/ClientBase.cpp:1063: DB::ClientBase::receiveAndProcessPacket(std::shared_ptr, bool) @ 0x0000000034b8caf3 in /clickhouse E 6. ./build_docker/./src/Client/ClientBase.cpp:1038: DB::ClientBase::receiveResult(std::shared_ptr, int, bool) @ 0x0000000034b8bfa6 in /clickhouse E 7. ./build_docker/./src/Client/ClientBase.cpp:0: DB::ClientBase::processOrdinaryQuery(String const&, std::shared_ptr) @ 0x0000000034b895f6 in /clickhouse E 8. ./build_docker/./src/Client/ClientBase.cpp:0: DB::ClientBase::processParsedSingleQuery(String const&, String const&, std::shared_ptr, std::optional, bool) @ 0x0000000034b850ea in /clickhouse E 9. ./build_docker/./src/Client/ClientBase.cpp:2053: DB::ClientBase::executeMultiQuery(String const&) @ 0x0000000034ba36a0 in /clickhouse E 10. ./build_docker/./src/Client/ClientBase.cpp:0: DB::ClientBase::processQueryText(String const&) @ 0x0000000034ba67b4 in /clickhouse E 11. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::ClientBase::runNonInteractive() @ 0x0000000034bb0d59 in /clickhouse E 12. ./build_docker/./programs/client/Client.cpp:0: DB::Client::main(std::vector> const&) @ 0x000000001da0265c in /clickhouse E 13. ./build_docker/./base/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x000000003dcf0b47 in /clickhouse E 14. ./build_docker/./programs/client/Client.cpp:0: mainEntryClickHouseClient(int, char**) @ 0x000000001da34dfc in /clickhouse E 15. ./build_docker/./programs/main.cpp:0: main @ 0x000000000bd6ed75 in /clickhouse E 16. ? @ 0x00007f6e4d16ed90 in ? E 17. ? @ 0x00007f6e4d16ee40 in ? E 18. _start @ 0x000000000bc95fee in /clickhouse E (version 23.8.16.42.altinitystable (altinity build)) E (query: SELECT * FROM mydb.filtered_table1) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:37:39 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:37:39 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:37:39 [ 373 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node (cluster.py:3423, query) 2024-10-21 22:37:39 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node2_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:37:39 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node2_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:37:39 [ 373 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node2 (cluster.py:3423, query) 2024-10-21 22:37:39 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) 2024-10-21 22:37:39 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table2 on node (cluster.py:3423, query) 2024-10-21 22:37:40 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table3 on node (cluster.py:3423, query) 2024-10-21 22:37:40 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:37:40 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:37:40 [ 373 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node (cluster.py:3423, query) 2024-10-21 22:37:40 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node2_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:37:40 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node2_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gQWxsb3dzIGFsbCByb3dzIC0tPgogICAgICAgICAgICA8ZGF0YWJhc2VzPgogICAgICAgICAgICAgICAgPG15ZGI+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMT4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj4xPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDxmaWx0ZXJlZF90YWJsZTI+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+MTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUyPgoKICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPjE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgIDwvbXlkYj4KICAgICAgICAgICAgPC9kYXRhYmFzZXM+CiAgICAgICAgPC9kZWZhdWx0PgogICAgPC91c2Vycz4KPC9jbGlja2hvdXNlPgo= | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:37:40 [ 373 ] DEBUG : Executing query SYSTEM RELOAD CONFIG on node2 (cluster.py:3423, query) 2024-10-21 22:37:40 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) ____________________________ test_dcl_introspection ____________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_dcl_introspection(): > assert node.query("SHOW POLICIES") == TSV( [ "another ON mydb.local", "default ON mydb.filtered_table1", "default ON mydb.filtered_table2", "default ON mydb.filtered_table3", ] ) test_row_policy/test.py:476: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.9.5:9000 (172.16.9.5:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e082673 in /clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5cce50 in /clickhouse E 2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x0000000034c1dca1 in /clickhouse E 3. ./build_docker/./src/Client/Connection.cpp:218: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x0000000034bfcedf in /clickhouse E 4. ./build_docker/./src/Client/Connection.cpp:423: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x0000000034c0377b in /clickhouse E 5. ./build_docker/./programs/client/Client.cpp:446: DB::Client::connect() @ 0x000000001da05a22 in /clickhouse E 6. ./build_docker/./programs/client/Client.cpp:0: DB::Client::main(std::vector> const&) @ 0x000000001da023be in /clickhouse E 7. ./build_docker/./base/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x000000003dcf0b47 in /clickhouse E 8. ./build_docker/./programs/client/Client.cpp:0: mainEntryClickHouseClient(int, char**) @ 0x000000001da34dfc in /clickhouse E 9. ./build_docker/./programs/main.cpp:0: main @ 0x000000000bd6ed75 in /clickhouse E 10. ? @ 0x00007f6435943d90 in ? E 11. ? @ 0x00007f6435943e40 in ? E 12. _start @ 0x000000000bc95fee in /clickhouse helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:37:57 [ 373 ] DEBUG : Executing query SHOW POLICIES on node (cluster.py:3423, query) _____________________________ test_dcl_management ______________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_dcl_management(): > copy_policy_xml("no_filters.xml") test_row_policy/test.py:603: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:08 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:08 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:08 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:08 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _________________ test_dcl_users_with_policies_from_users_xml __________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_dcl_users_with_policies_from_users_xml(): > node.query("CREATE USER X") test_row_policy/test.py:732: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.9.5:9000 (172.16.9.5:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e082673 in /clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5cce50 in /clickhouse E 2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x0000000034c1dca1 in /clickhouse E 3. ./build_docker/./src/Client/Connection.cpp:218: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x0000000034bfcedf in /clickhouse E 4. ./build_docker/./src/Client/Connection.cpp:423: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x0000000034c0377b in /clickhouse E 5. ./build_docker/./programs/client/Client.cpp:446: DB::Client::connect() @ 0x000000001da05a22 in /clickhouse E 6. ./build_docker/./programs/client/Client.cpp:0: DB::Client::main(std::vector> const&) @ 0x000000001da023be in /clickhouse E 7. ./build_docker/./base/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x000000003dcf0b47 in /clickhouse E 8. ./build_docker/./programs/client/Client.cpp:0: mainEntryClickHouseClient(int, char**) @ 0x000000001da34dfc in /clickhouse E 9. ./build_docker/./programs/main.cpp:0: main @ 0x000000000bd6ed75 in /clickhouse E 10. ? @ 0x00007f40fcd3cd90 in ? E 11. ? @ 0x00007f40fcd3ce40 in ? E 12. _start @ 0x000000000bc95fee in /clickhouse helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:08 [ 373 ] DEBUG : Executing query CREATE USER X on node (cluster.py:3423, query) _________________________ test_grant_create_row_policy _________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_grant_create_row_policy(): > copy_policy_xml("no_filters.xml") test_row_policy/test.py:635: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:19 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:19 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:19 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:19 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) ______________________________ test_introspection ______________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_introspection(): policies = [ [ "another ON mydb.local", "another", "mydb", "local", "5b23c389-7e18-06bf-a6bc-dd1afbbc0a97", "users_xml", "a = 1", 0, 0, "['another']", "[]", ], [ "default ON mydb.filtered_table1", "default", "mydb", "filtered_table1", "9e8a8f62-4965-2b5e-8599-57c7b99b3549", "users_xml", "a = 1", 0, 0, "['default']", "[]", ], [ "default ON mydb.filtered_table2", "default", "mydb", "filtered_table2", "cffae79d-b9bf-a2ef-b798-019c18470b25", "users_xml", "a + b < 1 or c - d > 5", 0, 0, "['default']", "[]", ], [ "default ON mydb.filtered_table3", "default", "mydb", "filtered_table3", "12fc5cef-e3da-3940-ec79-d8be3911f42b", "users_xml", "c = 1", 0, 0, "['default']", "[]", ], ] > assert node.query( "SELECT * from system.row_policies ORDER BY short_name, database, table" ) == TSV(policies) test_row_policy/test.py:470: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.9.5:9000 (172.16.9.5:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e082673 in /clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5cce50 in /clickhouse E 2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x0000000034c1dca1 in /clickhouse E 3. ./build_docker/./src/Client/Connection.cpp:218: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x0000000034bfcedf in /clickhouse E 4. ./build_docker/./src/Client/Connection.cpp:423: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x0000000034c0377b in /clickhouse E 5. ./build_docker/./programs/client/Client.cpp:446: DB::Client::connect() @ 0x000000001da05a22 in /clickhouse E 6. ./build_docker/./programs/client/Client.cpp:0: DB::Client::main(std::vector> const&) @ 0x000000001da023be in /clickhouse E 7. ./build_docker/./base/poco/Util/src/Application.cpp:0: Poco::Util::Application::run() @ 0x000000003dcf0b47 in /clickhouse E 8. ./build_docker/./programs/client/Client.cpp:0: mainEntryClickHouseClient(int, char**) @ 0x000000001da34dfc in /clickhouse E 9. ./build_docker/./programs/main.cpp:0: main @ 0x000000000bd6ed75 in /clickhouse E 10. ? @ 0x00007f6ff74b0d90 in ? E 11. ? @ 0x00007f6ff74b0e40 in ? E 12. _start @ 0x000000000bc95fee in /clickhouse helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:19 [ 373 ] DEBUG : Executing query SELECT * from system.row_policies ORDER BY short_name, database, table on node (cluster.py:3423, query) __________________________________ test_join ___________________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_join(): > assert node.query( "SELECT * FROM mydb.filtered_table1 as t1 ANY LEFT JOIN mydb.filtered_table1 as t2 ON t1.a = t2.b" ) == TSV([[1, 0, 1, 1], [1, 1, 1, 1]]) test_row_policy/test.py:126: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.9.5:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:30 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 as t1 ANY LEFT JOIN mydb.filtered_table1 as t2 ON t1.a = t2.b on node (cluster.py:3423, query) __________________________ test_miscellaneous_engines __________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_miscellaneous_engines(): > node.query( "CREATE ROW POLICY OR REPLACE pC ON mydb.other_table FOR SELECT USING a = 1 TO default" ) test_row_policy/test.py:814: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.9.5:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:34 [ 373 ] DEBUG : Executing query CREATE ROW POLICY OR REPLACE pC ON mydb.other_table FOR SELECT USING a = 1 TO default on node (cluster.py:3423, query) ____________ test_policy_from_users_xml_affects_only_user_assigned _____________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_policy_from_users_xml_affects_only_user_assigned(): > assert node.query("SELECT * FROM mydb.filtered_table1") == TSV([[1, 0], [1, 1]]) test_row_policy/test.py:208: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.9.5:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:37 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) __________________ test_policy_on_distributed_table_via_role ___________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_policy_on_distributed_table_via_role(): > node.query("DROP TABLE IF EXISTS local_tbl") test_row_policy/test.py:859: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.9.5:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:40 [ 373 ] DEBUG : Executing query DROP TABLE IF EXISTS local_tbl on node (cluster.py:3423, query) ________________________ test_reload_users_xml_by_timer ________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_reload_users_xml_by_timer(): > copy_policy_xml("normal_filters.xml") test_row_policy/test.py:381: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF...2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:43 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:43 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CiAgICA8dXNlcnM+CiAgICAgICAgPGRlZmF1bHQ+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8IS0tIFNpbXBsZSBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUxPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgPSAxPC9maWx0ZXI+CiAgICAgICAgICAgICAgICAgICAgPC9maWx0ZXJlZF90YWJsZTE+CgogICAgICAgICAgICAgICAgICAgIDwhLS0gQ29tcGxleCBleHByZXNzaW9uIGZpbHRlciAtLT4KICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyZWRfdGFibGUyPgogICAgICAgICAgICAgICAgICAgICAgICA8ZmlsdGVyPmEgKyBiICZsdDsgMSBvciBjIC0gZCAmZ3Q7IDU8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2ZpbHRlcmVkX3RhYmxlMj4KCiAgICAgICAgICAgICAgICAgICAgPCEtLSBGaWx0ZXIgd2l0aCBBTElBUyBjb2x1bW4gLS0+CiAgICAgICAgICAgICAgICAgICAgPGZpbHRlcmVkX3RhYmxlMz4KICAgICAgICAgICAgICAgICAgICAgICAgPGZpbHRlcj5jID0gMTwvZmlsdGVyPgogICAgICAgICAgICAgICAgICAgIDwvZmlsdGVyZWRfdGFibGUzPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2RlZmF1bHQ+CiAgICAgICAgPGFub3RoZXI+CiAgICAgICAgICAgIDwhLS0gRm9yIHRlc3RpbmcgdGhlIHRhYmxlIGZpbHRlcnMgLS0+CiAgICAgICAgICAgIDxkYXRhYmFzZXM+CiAgICAgICAgICAgICAgICA8bXlkYj4KICAgICAgICAgICAgICAgICAgICA8bG9jYWw+CiAgICAgICAgICAgICAgICAgICAgICAgIDxmaWx0ZXI+YSA9IDE8L2ZpbHRlcj4KICAgICAgICAgICAgICAgICAgICA8L2xvY2FsPgogICAgICAgICAgICAgICAgPC9teWRiPgogICAgICAgICAgICA8L2RhdGFiYXNlcz4KICAgICAgICA8L2Fub3RoZXI+CiAgICA8L3VzZXJzPgo8L2NsaWNraG91c2U+Cg== | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) _____________________ test_row_policy_filter_with_subquery _____________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_row_policy_filter_with_subquery(): > copy_policy_xml("no_filters.xml") test_row_policy/test.py:891: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_row_policy/test.py:36: in copy_policy_xml current_node.copy_file_to_container( helpers/cluster.py:3973: in copy_file_to_container return self.cluster.copy_file_to_container( helpers/cluster.py:2049: in copy_file_to_container self.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) out = res.stdout.decode("utf-8") err = res.stderr.decode("utf-8") # check_call(...) from subprocess does not print stderr, so we do it manually for outline in out.splitlines(): logging.debug(f"Stdout:{outline}") for errline in err.splitlines(): logging.debug(f"Stderr:{errline}") if res.returncode != 0: logging.debug(f"Exitcode:{res.returncode}") if env: logging.debug(f"Env:{env}") if not nothrow: > raise Exception( f"Command {args} return non-zero code {res.returncode}: {res.stderr.decode('utf-8')}" ) E Exception: Command ['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] return non-zero code 1: Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running helpers/cluster.py:121: Exception ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:43 [ 373 ] DEBUG : run container_id:roottestrowpolicy_node_1 detach:False nothrow:False cmd: ['bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:43 [ 373 ] DEBUG : Command:['docker', 'exec', 'roottestrowpolicy_node_1', 'bash', '-c', 'echo PGNsaWNraG91c2U+CjwvY2xpY2tob3VzZT4K | base64 --decode > /etc/clickhouse-server/users.d/row_policy.xml'] (cluster.py:105, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Stderr:Error response from daemon: Container c7a28220abc41fd9070651344163657c3887263c4a71afb1b84251bc3804d2ad is not running (cluster.py:115, run_and_check) 2024-10-21 22:38:43 [ 373 ] DEBUG : Exitcode:1 (cluster.py:117, run_and_check) __________________________________ test_smoke __________________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 def test_smoke(): > assert node.query("SELECT * FROM mydb.filtered_table1") == TSV([[1, 0], [1, 1]]) test_row_policy/test.py:98: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.9.5:9000). (NETWORK_ERROR) helpers/client.py:239: QueryRuntimeException ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:44 [ 373 ] DEBUG : Executing query SELECT * FROM mydb.filtered_table1 on node (cluster.py:3423, query) __________________________ test_encryption_functions ___________________________ [gw3] linux -- Python 3.10.12 /usr/bin/python3 def test_encryption_functions(): plaintext = new_password() cipher = new_password() key = new_password(32) iv8 = new_password(8) iv16 = new_password(16) add = new_password() encryption_functions = [ f"encrypt('aes-256-ofb', '{plaintext}', '{key}')", f"encrypt('aes-256-ofb', '{plaintext}', '{key}', '{iv16}')", f"encrypt('aes-256-gcm', '{plaintext}', '{key}', '{iv8}')", f"encrypt('aes-256-gcm', '{plaintext}', '{key}', '{iv8}', '{add}')", f"decrypt('aes-256-ofb', '{cipher}', '{key}', '{iv16}')", f"aes_encrypt_mysql('aes-256-ofb', '{plaintext}', '{key}', '{iv16}')", f"aes_decrypt_mysql('aes-256-ofb', '{cipher}', '{key}', '{iv16}')", f"tryDecrypt('aes-256-ofb', '{cipher}', '{key}', '{iv16}')", ] for encryption_function in encryption_functions: node.query(f"SELECT {encryption_function}") > check_logs( must_contain=[ "SELECT encrypt('aes-256-ofb', '[HIDDEN]')", "SELECT encrypt('aes-256-gcm', '[HIDDEN]')", "SELECT decrypt('aes-256-ofb', '[HIDDEN]')", "SELECT aes_encrypt_mysql('aes-256-ofb', '[HIDDEN]')", "SELECT aes_decrypt_mysql('aes-256-ofb', '[HIDDEN]')", "SELECT tryDecrypt('aes-256-ofb', '[HIDDEN]')", ], must_not_contain=[plaintext, cipher, key, iv8, iv16, add], ) test_mask_sensitive_info/test.py:381: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ test_mask_sensitive_info/test.py:46: in check_logs assert not node.contains_in_log(escaped_str) helpers/cluster.py:3863: in contains_in_log result = self.exec_in_container( helpers/cluster.py:3840: in exec_in_container return self.cluster.exec_in_container( helpers/cluster.py:2012: in exec_in_container result = subprocess_check_call( helpers/cluster.py:158: in subprocess_check_call return run_and_check(args, detach=detach, nothrow=nothrow) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "ZXQ1RNKHSV3J4PT6" /var/log/clickhouse-server/clickhouse-server.log || true'] env = None, shell = False, stdout = -1, stderr = -1, timeout = 300 nothrow = False, detach = False def run_and_check( args, env=None, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=300, nothrow=False, detach=False, ): if detach: subprocess.Popen( args, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, env=env, shell=shell, ) return logging.debug(f"Command:{args}") res = subprocess.run( args, stdout=stdout, stderr=stderr, env=env, shell=shell, timeout=timeout ) > out = res.stdout.decode("utf-8") E UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 153: invalid start byte helpers/cluster.py:109: UnicodeDecodeError ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:51 [ 379 ] DEBUG : Executing query SELECT encrypt('aes-256-ofb', 'ZXQ1RNKHSV3J4PT6', 'TXEYEU9TQWN5MJL9MZUJ949Z8997KP98') on node (cluster.py:3423, query) 2024-10-21 22:38:51 [ 379 ] DEBUG : Executing query SELECT encrypt('aes-256-ofb', 'ZXQ1RNKHSV3J4PT6', 'TXEYEU9TQWN5MJL9MZUJ949Z8997KP98', '2GYQRXBLWND0U624') on node (cluster.py:3423, query) 2024-10-21 22:38:51 [ 379 ] DEBUG : Executing query SELECT encrypt('aes-256-gcm', 'ZXQ1RNKHSV3J4PT6', 'TXEYEU9TQWN5MJL9MZUJ949Z8997KP98', 'I72OAVR9') on node (cluster.py:3423, query) 2024-10-21 22:38:51 [ 379 ] DEBUG : Executing query SELECT encrypt('aes-256-gcm', 'ZXQ1RNKHSV3J4PT6', 'TXEYEU9TQWN5MJL9MZUJ949Z8997KP98', 'I72OAVR9', '4KS5TY66Q63CTG80') on node (cluster.py:3423, query) 2024-10-21 22:38:52 [ 379 ] DEBUG : Executing query SELECT decrypt('aes-256-ofb', 'USFB6HQ3V6T7RK5Y', 'TXEYEU9TQWN5MJL9MZUJ949Z8997KP98', '2GYQRXBLWND0U624') on node (cluster.py:3423, query) 2024-10-21 22:38:52 [ 379 ] DEBUG : Executing query SELECT aes_encrypt_mysql('aes-256-ofb', 'ZXQ1RNKHSV3J4PT6', 'TXEYEU9TQWN5MJL9MZUJ949Z8997KP98', '2GYQRXBLWND0U624') on node (cluster.py:3423, query) 2024-10-21 22:38:52 [ 379 ] DEBUG : Executing query SELECT aes_decrypt_mysql('aes-256-ofb', 'USFB6HQ3V6T7RK5Y', 'TXEYEU9TQWN5MJL9MZUJ949Z8997KP98', '2GYQRXBLWND0U624') on node (cluster.py:3423, query) 2024-10-21 22:38:52 [ 379 ] DEBUG : Executing query SELECT tryDecrypt('aes-256-ofb', 'USFB6HQ3V6T7RK5Y', 'TXEYEU9TQWN5MJL9MZUJ949Z8997KP98', '2GYQRXBLWND0U624') on node (cluster.py:3423, query) 2024-10-21 22:38:52 [ 379 ] DEBUG : Executing query SYSTEM FLUSH LOGS on node (cluster.py:3423, query) 2024-10-21 22:38:53 [ 379 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT encrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:53 [ 379 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT encrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.10.21 22:38:51.338567 [ 8 ] {673ea6e4-821a-464e-96d1-5d845e5d9e57} executeQuery: (from 172.16.4.1:55784) SELECT encrypt('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.10.21 22:38:51.532305 [ 8 ] {9095b324-2637-4f66-b66d-e6ecc0121f94} executeQuery: (from 172.16.4.1:55786) SELECT encrypt('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT encrypt(\'aes-256-gcm\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:53 [ 379 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT encrypt(\'aes-256-gcm\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.10.21 22:38:51.761693 [ 8 ] {3ba74824-12d7-40c1-95f0-4b864590f284} executeQuery: (from 172.16.4.1:55796) SELECT encrypt('aes-256-gcm', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.10.21 22:38:51.961150 [ 8 ] {a3a15640-1774-4f61-8513-bde7958086ff} executeQuery: (from 172.16.4.1:55810) SELECT encrypt('aes-256-gcm', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT decrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:53 [ 379 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT decrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.10.21 22:38:52.178602 [ 8 ] {e1ae14ca-abf9-4e3a-91de-9ad71cc180fa} executeQuery: (from 172.16.4.1:55824) SELECT decrypt('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT aes_encrypt_mysql(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:53 [ 379 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT aes_encrypt_mysql(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.10.21 22:38:52.400133 [ 8 ] {cbc2d8ec-5a84-43c0-8e87-44622f52fe03} executeQuery: (from 172.16.4.1:55826) SELECT aes_encrypt_mysql('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT aes_decrypt_mysql(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:53 [ 379 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT aes_decrypt_mysql(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.10.21 22:38:52.621970 [ 8 ] {1d849a7f-0454-496e-be7f-7b422d5def71} executeQuery: (from 172.16.4.1:55834) SELECT aes_decrypt_mysql('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT tryDecrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:53 [ 379 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "SELECT tryDecrypt(\'aes-256-ofb\', \'\\[HIDDEN\\]\')" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2024.10.21 22:38:52.841753 [ 8 ] {897e2303-1d74-4da6-a5eb-ad3be74b5c41} executeQuery: (from 172.16.4.1:55840) SELECT tryDecrypt('aes-256-ofb', '[HIDDEN]') (stage: Complete) (cluster.py:113, run_and_check) 2024-10-21 22:38:53 [ 379 ] DEBUG : run container_id:roottestmasksensitiveinfo_node_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "ZXQ1RNKHSV3J4PT6" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:2004, exec_in_container) 2024-10-21 22:38:53 [ 379 ] DEBUG : Command:['docker', 'exec', 'roottestmasksensitiveinfo_node_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "ZXQ1RNKHSV3J4PT6" /var/log/clickhouse-server/clickhouse-server.log || true'] (cluster.py:105, run_and_check) _______________________ test_default_database_on_cluster _______________________ [gw9] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_default_database_on_cluster(started_cluster): ch1.query( database="test_default_database", sql="CREATE TABLE test_local_table ENGINE MergeTree PARTITION BY i ORDER BY tuple() SETTINGS max_partitions_to_read = 1 AS SELECT arrayJoin([1, 2]) i;", ) > assert ( ch2.query( sql="SELECT * FROM remote('ch1:9000', test_default_database, test_local_table) ORDER BY i FORMAT TSV SETTINGS max_partitions_to_read = 0;", ) == "1\n2\n" ) test_passing_max_partitions_to_read_remotely/test.py:26: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3424: in query return self.client.query( helpers/client.py:36: in wrap return func(self, *args, **kwargs) helpers/client.py:74: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 81, stderr: Received exception from server (version 23.8.16): E Code: 81. DB::Exception: Received from 172.16.23.2:9000. DB::Exception: Database test_default_database does not exist. Stack trace: E E 0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x000000003e082673 in /usr/bin/clickhouse E 1. ./build_docker/./src/Common/Exception.cpp:98: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d5cce50 in /usr/bin/clickhouse E 2. DB::Exception::Exception(int, FormatStringHelperImpl::type>, String&&) @ 0x000000000bd8589b in /usr/bin/clickhouse E 3. ./build_docker/./src/Interpreters/DatabaseCatalog.cpp:650: DB::DatabaseCatalog::getDatabase(String const&) const @ 0x000000002fb33702 in /usr/bin/clickhouse E 4. ./build_docker/./src/Interpreters/Context.cpp:0: DB::Context::resolveStorageID(DB::StorageID, DB::Context::StorageNamespace) const @ 0x000000002f990641 in /usr/bin/clickhouse E 5. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: DB::StorageDistributed::read(DB::QueryPlan&, std::vector> const&, std::shared_ptr const&, DB::SelectQueryInfo&, std::shared_ptr, DB::QueryProcessingStage::Enum, unsigned long, unsigned long) @ 0x0000000032de2c94 in /usr/bin/clickhouse E 6. ./build_docker/./src/Planner/PlannerJoinTree.cpp:756: DB::(anonymous namespace)::buildQueryPlanForTableExpression(std::shared_ptr, DB::SelectQueryInfo const&, DB::SelectQueryOptions const&, std::shared_ptr&, bool, bool) @ 0x0000000031666617 in /usr/bin/clickhouse E 7. ./build_docker/./src/Planner/PlannerJoinTree.cpp:1450: DB::buildJoinTreeQueryPlan(std::shared_ptr const&, DB::SelectQueryInfo const&, DB::SelectQueryOptions&, std::unordered_set, std::equal_to, std::allocator> const&, std::shared_ptr&) @ 0x000000003165712a in /usr/bin/clickhouse E 8. ./build_docker/./src/Planner/Planner.cpp:1258: DB::Planner::buildPlanForQueryNode() @ 0x0000000031608256 in /usr/bin/clickhouse E 9. ./build_docker/./src/Planner/Planner.cpp:1090: DB::Planner::buildQueryPlanIfNeeded() @ 0x00000000316025c7 in /usr/bin/clickhouse E 10. ./build_docker/./src/Planner/Planner.h:44: DB::InterpreterSelectQueryAnalyzer::getQueryPlan() @ 0x0000000031600212 in /usr/bin/clickhouse E 11. ./build_docker/./src/Interpreters/executeQuery.cpp:1041: DB::executeQueryImpl(char const*, char const*, std::shared_ptr, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x00000000320ede1d in /usr/bin/clickhouse E 12. ./build_docker/./src/Interpreters/executeQuery.cpp:1224: DB::executeQuery(String const&, std::shared_ptr, bool, DB::QueryProcessingStage::Enum) @ 0x00000000320e5aa1 in /usr/bin/clickhouse E 13. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000034ed6d3d in /usr/bin/clickhouse E 14. ./build_docker/./src/Server/TCPHandler.cpp:2161: DB::TCPHandler::run() @ 0x0000000034f13b30 in /usr/bin/clickhouse E 15. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x000000003dcb69cf in /usr/bin/clickhouse E 16. ./build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48: Poco::Net::TCPServerDispatcher::run() @ 0x000000003dcb7797 in /usr/bin/clickhouse E 17. ./build_docker/./base/poco/Foundation/src/ThreadPool.cpp:0: Poco::PooledThread::run() @ 0x000000003e196fbc in /usr/bin/clickhouse E 18. ./build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:0: Poco::ThreadImpl::runnableEntry(void*) @ 0x000000003e190068 in /usr/bin/clickhouse E 19. ? @ 0x00007fb82ff9eac3 in ? E 20. ? @ 0x00007fb830030850 in ? E . (UNKNOWN_DATABASE) E (query: SELECT * FROM remote('ch1:9000', test_default_database, test_local_table) ORDER BY i FORMAT TSV SETTINGS max_partitions_to_read = 0;) helpers/client.py:239: QueryRuntimeException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ------------------------------ Captured log setup ------------------------------ 2024-10-21 22:38:46 [ 397 ] INFO : Running tests in /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/test.py (cluster.py:2637, start) 2024-10-21 22:38:46 [ 397 ] DEBUG : Cluster start called. is_up=False (cluster.py:2644, start) 2024-10-21 22:38:46 [ 397 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 27 Running: 21 Paused: 0 Stopped: 6 Images: 41 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: 2110fc98d269 ID: f2e62ed7-a1b0-4a54-9efb-9f1d0b410eca Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-10-21 22:38:46 [ 397 ] DEBUG : Docker networks for project roottestpassingmaxpartitionstoreadremotely are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-10-21 22:38:46 [ 397 ] DEBUG : Docker containers for project roottestpassingmaxpartitionstoreadremotely are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-10-21 22:38:46 [ 397 ] DEBUG : Docker volumes for project roottestpassingmaxpartitionstoreadremotely are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-10-21 22:38:46 [ 397 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-10-21 22:38:46 [ 397 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 27 Running: 21 Paused: 0 Stopped: 6 Images: 41 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: 2110fc98d269 ID: f2e62ed7-a1b0-4a54-9efb-9f1d0b410eca Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-10-21 22:38:46 [ 397 ] DEBUG : Docker networks for project roottestpassingmaxpartitionstoreadremotely are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-10-21 22:38:46 [ 397 ] DEBUG : Docker containers for project roottestpassingmaxpartitionstoreadremotely are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-10-21 22:38:46 [ 397 ] DEBUG : Docker volumes for project roottestpassingmaxpartitionstoreadremotely are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-10-21 22:38:46 [ 397 ] DEBUG : Command:docker container list --all --filter name='^/roottestpassingmaxpartitionstoreadremotely_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-10-21 22:38:46 [ 397 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-10-21 22:38:46 [ 397 ] DEBUG : No running containers for project: roottestpassingmaxpartitionstoreadremotely (cluster.py:782, cleanup) 2024-10-21 22:38:46 [ 397 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-10-21 22:38:46 [ 397 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-10-21 22:38:46 [ 397 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-10-21 22:38:46 [ 397 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-10-21 22:38:46 [ 397 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-10-21 22:38:46 [ 397 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-10-21 22:38:46 [ 397 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-10-21 22:38:46 [ 397 ] DEBUG : Stdout:3 (cluster.py:113, run_and_check) 2024-10-21 22:38:46 [ 397 ] DEBUG : Setup directory for instance: ch1 (cluster.py:2657, start) 2024-10-21 22:38:46 [ 397 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/configs/config.d (cluster.py:4383, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/database (cluster.py:4400, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/logs (cluster.py:4411, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Setup directory for instance: ch2 (cluster.py:2657, start) 2024-10-21 22:38:46 [ 397 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/configs/config.d (cluster.py:4383, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/database (cluster.py:4400, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/logs (cluster.py:4411, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir) 2024-10-21 22:38:46 [ 397 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env (cluster.py:78, _create_env_file) 2024-10-21 22:38:46 [ 397 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-10-21 22:38:46 [ 397 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-10-21 22:38:46 [ 397 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-10-21 22:38:46 [ 397 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-10-21 22:38:46 [ 397 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:546, _make_request) 2024-10-21 22:38:46 [ 397 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env', '--project-name', 'roottestpassingmaxpartitionstoreadremotely', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/docker-compose.yml', 'pull'] (cluster.py:105, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : Stderr:Pulling ch2 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : Stderr:Pulling ch1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : Stderr:Pulling ch1 ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : Stderr:Pulling ch2 ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : Stderr:Pulling ch1 ... digest: sha256:fe9d6877285f6e4ecd... (cluster.py:115, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : Stderr:Pulling ch1 ... status: image is up to date for a... (cluster.py:115, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : Stderr:Pulling ch1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : Stderr:Pulling ch2 ... digest: sha256:fe9d6877285f6e4ecd... (cluster.py:115, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : Stderr:Pulling ch2 ... status: image is up to date for a... (cluster.py:115, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : Stderr:Pulling ch2 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:57 [ 397 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env --project-name roottestpassingmaxpartitionstoreadremotely --file /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/docker-compose.yml --file /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/docker-compose.yml up -d --no-recreate') (cluster.py:2980, start) 2024-10-21 22:38:57 [ 397 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env', '--project-name', 'roottestpassingmaxpartitionstoreadremotely', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:105, run_and_check) 2024-10-21 22:38:59 [ 397 ] DEBUG : Stderr:Creating network "roottestpassingmaxpartitionstoreadremotely_default" with the default driver (cluster.py:115, run_and_check) 2024-10-21 22:38:59 [ 397 ] DEBUG : Stderr:Creating roottestpassingmaxpartitionstoreadremotely_ch1_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:59 [ 397 ] DEBUG : Stderr:Creating roottestpassingmaxpartitionstoreadremotely_ch2_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:59 [ 397 ] DEBUG : Stderr:Creating roottestpassingmaxpartitionstoreadremotely_ch2_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:59 [ 397 ] DEBUG : Stderr:Creating roottestpassingmaxpartitionstoreadremotely_ch1_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:59 [ 397 ] DEBUG : ClickHouse instance created (cluster.py:2988, start) 2024-10-21 22:38:59 [ 397 ] DEBUG : get_instance_ip instance_name=ch1 (cluster.py:1971, get_instance_ip) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestpassingmaxpartitionstoreadremotely_ch1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:59 [ 397 ] DEBUG : Waiting for ClickHouse start in ch1, ip: 172.16.23.3... (cluster.py:2995, start) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestpassingmaxpartitionstoreadremotely_ch1_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:59 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:00 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:00 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:00 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:00 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:00 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:00 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:00 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:00 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:00 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:00 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:01 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:01 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:01 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/e237308763d737957e68c619f48d1722c28cbee591df73317ccaa5a23b82a384/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:01 [ 397 ] DEBUG : ClickHouse ch1 started (cluster.py:2999, start) 2024-10-21 22:39:01 [ 397 ] DEBUG : get_instance_ip instance_name=ch2 (cluster.py:1971, get_instance_ip) 2024-10-21 22:39:01 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestpassingmaxpartitionstoreadremotely_ch2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:01 [ 397 ] DEBUG : Waiting for ClickHouse start in ch2, ip: 172.16.23.2... (cluster.py:2995, start) 2024-10-21 22:39:01 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestpassingmaxpartitionstoreadremotely_ch2_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:01 [ 397 ] DEBUG : http://localhost:None "GET /v1.42/containers/96adbd588eb73682af5c884285a3004ae13f63541100e94366006b7b53edcac1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:39:01 [ 397 ] DEBUG : ClickHouse ch2 started (cluster.py:2999, start) 2024-10-21 22:39:01 [ 397 ] DEBUG : Executing query CREATE DATABASE test_default_database; on ch1 (cluster.py:3423, query) ------------------------------ Captured log call ------------------------------- 2024-10-21 22:39:01 [ 397 ] DEBUG : Executing query CREATE TABLE test_local_table ENGINE MergeTree PARTITION BY i ORDER BY tuple() SETTINGS max_partitions_to_read = 1 AS SELECT arrayJoin([1, 2]) i; on ch1 (cluster.py:3423, query) 2024-10-21 22:39:01 [ 397 ] DEBUG : Executing query SELECT * FROM remote('ch1:9000', test_default_database, test_local_table) ORDER BY i FORMAT TSV SETTINGS max_partitions_to_read = 0; on ch2 (cluster.py:3423, query) --------------------------- Captured stderr teardown --------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ---------------------------- Captured log teardown ----------------------------- 2024-10-21 22:39:02 [ 397 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env', '--project-name', 'roottestpassingmaxpartitionstoreadremotely', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:105, run_and_check) 2024-10-21 22:39:05 [ 397 ] DEBUG : Stderr:Stopping roottestpassingmaxpartitionstoreadremotely_ch2_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:39:05 [ 397 ] DEBUG : Stderr:Stopping roottestpassingmaxpartitionstoreadremotely_ch1_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:39:05 [ 397 ] DEBUG : Stderr:Stopping roottestpassingmaxpartitionstoreadremotely_ch2_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:39:05 [ 397 ] DEBUG : Stderr:Stopping roottestpassingmaxpartitionstoreadremotely_ch1_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:39:05 [ 397 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-10-21 22:39:05 [ 397 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/logs/stderr.log* || true'] (cluster.py:105, run_and_check) 2024-10-21 22:39:05 [ 397 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/.env', '--project-name', 'roottestpassingmaxpartitionstoreadremotely', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_passing_max_partitions_to_read_remotely/_instances_0/ch2/docker-compose.yml', 'down', '--volumes'] (cluster.py:105, run_and_check) 2024-10-21 22:39:06 [ 397 ] DEBUG : Stderr:Removing roottestpassingmaxpartitionstoreadremotely_ch2_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:39:06 [ 397 ] DEBUG : Stderr:Removing roottestpassingmaxpartitionstoreadremotely_ch1_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:39:06 [ 397 ] DEBUG : Stderr:Removing roottestpassingmaxpartitionstoreadremotely_ch2_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:39:06 [ 397 ] DEBUG : Stderr:Removing roottestpassingmaxpartitionstoreadremotely_ch1_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:39:06 [ 397 ] DEBUG : Stderr:Removing network roottestpassingmaxpartitionstoreadremotely_default (cluster.py:115, run_and_check) 2024-10-21 22:39:06 [ 397 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-10-21 22:39:06 [ 397 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 12 Running: 12 Paused: 0 Stopped: 0 Images: 41 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: 2110fc98d269 ID: f2e62ed7-a1b0-4a54-9efb-9f1d0b410eca Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-10-21 22:39:06 [ 397 ] DEBUG : Docker networks for project roottestpassingmaxpartitionstoreadremotely are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-10-21 22:39:06 [ 397 ] DEBUG : Docker containers for project roottestpassingmaxpartitionstoreadremotely are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-10-21 22:39:06 [ 397 ] DEBUG : Docker volumes for project roottestpassingmaxpartitionstoreadremotely are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-10-21 22:39:06 [ 397 ] DEBUG : Command:docker container list --all --filter name='^/roottestpassingmaxpartitionstoreadremotely_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-10-21 22:39:06 [ 397 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-10-21 22:39:06 [ 397 ] DEBUG : No running containers for project: roottestpassingmaxpartitionstoreadremotely (cluster.py:782, cleanup) 2024-10-21 22:39:06 [ 397 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-10-21 22:39:06 [ 397 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-10-21 22:39:06 [ 397 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-10-21 22:39:06 [ 397 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-10-21 22:39:06 [ 397 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-10-21 22:39:06 [ 397 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-10-21 22:39:06 [ 397 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-10-21 22:39:06 [ 397 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check) __________________ test_mutations_with_merge_background_task ___________________ [gw7] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_mutations_with_merge_background_task(started_cluster): instance_test_mutations.query( """SYSTEM STOP MERGES test_mutations_with_ast_elements""" ) ## The number of asts per query is 15 for execution_times_for_mutation in range(100): instance_test_mutations.query( """ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1)""" ) all_done = False for wait_times_for_mutation in range(100): # wait for replication 80 seconds max time.sleep(0.8) def get_done_mutations(instance): instance_test_mutations.query( """DETACH TABLE test_mutations_with_ast_elements""" ) instance_test_mutations.query( """ATTACH TABLE test_mutations_with_ast_elements""" ) return int( instance.query( "SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0" ).rstrip() ) if get_done_mutations(instance_test_mutations) == 100: all_done = True break print( instance_test_mutations.query( "SELECT mutation_id, command, parts_to_do, is_done FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 FORMAT TSVWithNames" ) ) > assert all_done E assert False test_mutations_with_merge_tree/test.py:67: AssertionError ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ---------------------------- Captured stderr setup ----------------------------- WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ WARNING: API is accessible on http://0.0.0.0:2375 without encryption. Access to the remote API is equivalent to root access on the host. Refer to the 'Docker daemon attack surface' section in the documentation for more information: https://docs.docker.com/go/attack-surface/ ------------------------------ Captured log setup ------------------------------ 2024-10-21 22:37:57 [ 391 ] INFO : Running tests in /ClickHouse/tests/integration/test_mutations_with_merge_tree/test.py (cluster.py:2637, start) 2024-10-21 22:37:57 [ 391 ] DEBUG : Cluster start called. is_up=False (cluster.py:2644, start) 2024-10-21 22:37:57 [ 391 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 27 Running: 26 Paused: 0 Stopped: 1 Images: 41 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: 2110fc98d269 ID: f2e62ed7-a1b0-4a54-9efb-9f1d0b410eca Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-10-21 22:37:57 [ 391 ] DEBUG : Docker networks for project roottestmutationswithmergetree are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-10-21 22:37:57 [ 391 ] DEBUG : Docker containers for project roottestmutationswithmergetree are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-10-21 22:37:57 [ 391 ] DEBUG : Docker volumes for project roottestmutationswithmergetree are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-10-21 22:37:57 [ 391 ] DEBUG : Cleanup called (cluster.py:754, cleanup) 2024-10-21 22:37:57 [ 391 ] DEBUG : !!! Docker info: Client: Docker Engine - Community Version: 27.3.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.17.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.29.7 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 27 Running: 26 Paused: 0 Stopped: 1 Images: 41 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c runc version: v1.1.14-0-g2c9f560 init version: de40ad0 Security Options: seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-122-generic Operating System: Ubuntu 22.04.5 LTS (containerized) OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 30.6GiB Name: 2110fc98d269 ID: f2e62ed7-a1b0-4a54-9efb-9f1d0b410eca Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 65.108.242.32:5000 127.0.0.0/8 Registry Mirrors: http://65.108.242.32:5000/ Live Restore Enabled: false Default Address Pools: Base: 172.17.0.0/12, Size: 24 (cluster.py:722, print_all_docker_pieces) 2024-10-21 22:37:57 [ 391 ] DEBUG : Docker networks for project roottestmutationswithmergetree are NETWORK ID NAME DRIVER SCOPE (cluster.py:733, print_all_docker_pieces) 2024-10-21 22:37:57 [ 391 ] DEBUG : Docker containers for project roottestmutationswithmergetree are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:741, print_all_docker_pieces) 2024-10-21 22:37:57 [ 391 ] DEBUG : Docker volumes for project roottestmutationswithmergetree are DRIVER VOLUME NAME (cluster.py:749, print_all_docker_pieces) 2024-10-21 22:37:57 [ 391 ] DEBUG : Command:docker container list --all --filter name='^/roottestmutationswithmergetree_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:105, run_and_check) 2024-10-21 22:37:57 [ 391 ] DEBUG : Unstopped containers: {} (cluster.py:768, cleanup) 2024-10-21 22:37:57 [ 391 ] DEBUG : No running containers for project: roottestmutationswithmergetree (cluster.py:782, cleanup) 2024-10-21 22:37:57 [ 391 ] DEBUG : Trying to prune unused networks... (cluster.py:788, cleanup) 2024-10-21 22:37:57 [ 391 ] DEBUG : Trying to prune unused images... (cluster.py:804, cleanup) 2024-10-21 22:37:57 [ 391 ] DEBUG : Command:['docker', 'image', 'prune', '-f'] (cluster.py:105, run_and_check) 2024-10-21 22:37:57 [ 391 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:113, run_and_check) 2024-10-21 22:37:57 [ 391 ] DEBUG : Images pruned (cluster.py:807, cleanup) 2024-10-21 22:37:57 [ 391 ] DEBUG : Trying to prune unused volumes... (cluster.py:813, cleanup) 2024-10-21 22:37:57 [ 391 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:105, run_and_check) 2024-10-21 22:37:57 [ 391 ] DEBUG : Stdout:1 (cluster.py:113, run_and_check) 2024-10-21 22:37:57 [ 391 ] DEBUG : Setup directory for instance: test_mutations_with_merge_tree (cluster.py:2657, start) 2024-10-21 22:37:57 [ 391 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4309, create_dir) 2024-10-21 22:37:57 [ 391 ] DEBUG : Create directory for common tests configuration (cluster.py:4314, create_dir) 2024-10-21 22:37:57 [ 391 ] DEBUG : Copy common configuration from helpers (cluster.py:4334, create_dir) 2024-10-21 22:37:57 [ 391 ] DEBUG : Generate and write macros file (cluster.py:4353, create_dir) 2024-10-21 22:37:57 [ 391 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_mutations_with_merge_tree/configs/config.xml'] to /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/configs/config.d (cluster.py:4383, create_dir) 2024-10-21 22:37:57 [ 391 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/database (cluster.py:4400, create_dir) 2024-10-21 22:37:57 [ 391 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/logs (cluster.py:4411, create_dir) 2024-10-21 22:37:57 [ 391 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4487, create_dir) 2024-10-21 22:37:57 [ 391 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/.env (cluster.py:78, _create_env_file) 2024-10-21 22:37:57 [ 391 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-10-21 22:37:57 [ 391 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-10-21 22:37:57 [ 391 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2024-10-21 22:37:57 [ 391 ] DEBUG : No config file found (config.py:28, find_config_file) 2024-10-21 22:37:57 [ 391 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:546, _make_request) 2024-10-21 22:37:57 [ 391 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/.env', '--project-name', 'roottestmutationswithmergetree', '--file', '/ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/docker-compose.yml', 'pull'] (cluster.py:105, run_and_check) 2024-10-21 22:38:07 [ 391 ] DEBUG : Stderr:Pulling test_mutations_with_merge_tree ... (cluster.py:115, run_and_check) 2024-10-21 22:38:07 [ 391 ] DEBUG : Stderr:Pulling test_mutations_with_merge_tree ... pulling from altinityinfra/integr... (cluster.py:115, run_and_check) 2024-10-21 22:38:07 [ 391 ] DEBUG : Stderr:Pulling test_mutations_with_merge_tree ... digest: sha256:fe9d6877285f6e4ecd... (cluster.py:115, run_and_check) 2024-10-21 22:38:07 [ 391 ] DEBUG : Stderr:Pulling test_mutations_with_merge_tree ... status: image is up to date for a... (cluster.py:115, run_and_check) 2024-10-21 22:38:07 [ 391 ] DEBUG : Stderr:Pulling test_mutations_with_merge_tree ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:07 [ 391 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/.env --project-name roottestmutationswithmergetree --file /ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/docker-compose.yml up -d --no-recreate') (cluster.py:2980, start) 2024-10-21 22:38:07 [ 391 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/.env', '--project-name', 'roottestmutationswithmergetree', '--file', '/ClickHouse/tests/integration/test_mutations_with_merge_tree/_instances_0/test_mutations_with_merge_tree/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:105, run_and_check) 2024-10-21 22:38:08 [ 391 ] DEBUG : Stderr:Creating network "roottestmutationswithmergetree_default" with the default driver (cluster.py:115, run_and_check) 2024-10-21 22:38:08 [ 391 ] DEBUG : Stderr:Creating roottestmutationswithmergetree_test_mutations_with_merge_tree_1 ... (cluster.py:115, run_and_check) 2024-10-21 22:38:08 [ 391 ] DEBUG : Stderr:Creating roottestmutationswithmergetree_test_mutations_with_merge_tree_1 ... done (cluster.py:115, run_and_check) 2024-10-21 22:38:08 [ 391 ] DEBUG : ClickHouse instance created (cluster.py:2988, start) 2024-10-21 22:38:08 [ 391 ] DEBUG : get_instance_ip instance_name=test_mutations_with_merge_tree (cluster.py:1971, get_instance_ip) 2024-10-21 22:38:08 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestmutationswithmergetree_test_mutations_with_merge_tree_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:08 [ 391 ] DEBUG : Waiting for ClickHouse start in test_mutations_with_merge_tree, ip: 172.16.13.2... (cluster.py:2995, start) 2024-10-21 22:38:08 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestmutationswithmergetree_test_mutations_with_merge_tree_1/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:08 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:08 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:08 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:08 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:08 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:08 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:08 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:09 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:09 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:09 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:09 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:09 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:09 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:09 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:09 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:09 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:09 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:10 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:10 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:10 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:10 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:10 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:10 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:10 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:10 [ 391 ] DEBUG : http://localhost:None "GET /v1.42/containers/2475995e719f7119751a9f38a8bf97bdda437481398f9338b3a7eae0d48f7e95/json HTTP/1.1" 200 None (connectionpool.py:546, _make_request) 2024-10-21 22:38:10 [ 391 ] DEBUG : ClickHouse test_mutations_with_merge_tree started (cluster.py:2999, start) 2024-10-21 22:38:10 [ 391 ] DEBUG : Executing query CREATE TABLE test_mutations_with_ast_elements(date Date, a UInt64, b String) ENGINE = MergeTree PARTITION BY toYYYYMM(date) ORDER BY (a, date) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:11 [ 391 ] DEBUG : Executing query INSERT INTO test_mutations_with_ast_elements SELECT '2019-07-29' AS date, 1, toString(number) FROM numbers(1) SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) ----------------------------- Captured stdout call ----------------------------- mutation_id command parts_to_do is_done mutation_2.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_3.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_4.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_5.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_6.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_7.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_8.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_9.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_10.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_11.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_12.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_13.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_14.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_15.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_16.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_17.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_18.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_19.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_20.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_21.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_22.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_23.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_24.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_25.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_26.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_27.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_28.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_29.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_30.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_31.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_32.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_33.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_34.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_35.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_36.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_37.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_38.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_39.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_40.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_41.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_42.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_43.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_44.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_45.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_46.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_47.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_48.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_49.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_50.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_51.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_52.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_53.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_54.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_55.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_56.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_57.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_58.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_59.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_60.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_61.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_62.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_63.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_64.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_65.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_66.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_67.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_68.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_69.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_70.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_71.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_72.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_73.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_74.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_75.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_76.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_77.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_78.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_79.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_80.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_81.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_82.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_83.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_84.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_85.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_86.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_87.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_88.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_89.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_90.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_91.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_92.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_93.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_94.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_95.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_96.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_97.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_98.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_99.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_100.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 mutation_101.txt DELETE WHERE (1 = 1) AND (toUInt32(b) IN (1)) 1 0 ------------------------------ Captured log call ------------------------------- 2024-10-21 22:38:11 [ 391 ] DEBUG : Executing query SYSTEM STOP MERGES test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:11 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:11 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:11 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:12 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:12 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:12 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:12 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:12 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:13 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:13 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:13 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:13 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:13 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:13 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:14 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:14 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:14 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:14 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:14 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:14 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:15 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:15 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:15 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:15 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:15 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:16 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:16 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:16 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:16 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:17 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:17 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:17 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:17 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:17 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:18 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:18 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:18 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:18 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:18 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:19 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:19 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:19 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:20 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:20 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:20 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:20 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:20 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:21 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:21 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:21 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:22 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:22 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:22 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:23 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:23 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:23 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:23 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:23 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:23 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:24 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:24 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:24 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:24 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:24 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:25 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:25 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:25 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:25 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:26 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:26 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:26 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:26 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:26 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:27 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:27 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:27 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:27 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:27 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:27 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:28 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:28 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:28 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:28 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:29 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:29 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:29 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:29 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:29 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:30 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:30 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:30 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:30 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:30 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:31 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:31 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:31 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:31 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:31 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:31 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:32 [ 391 ] DEBUG : Executing query ALTER TABLE test_mutations_with_ast_elements DELETE WHERE 1 = 1 AND toUInt32(b) IN (1) on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:33 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:33 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:33 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:34 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:34 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:35 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:36 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:36 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:36 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:37 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:37 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:37 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:39 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:39 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:39 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:40 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:40 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:40 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:41 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:42 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:42 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:43 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:43 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:44 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:45 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:45 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:45 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:46 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:46 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:47 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:48 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:48 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:48 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:50 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:50 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:50 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:51 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:51 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:51 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:53 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:53 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:53 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:54 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:54 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:54 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:55 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:56 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:56 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:57 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:57 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:57 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:59 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:59 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:38:59 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:00 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:00 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:00 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:01 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:02 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:02 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:03 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:03 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:03 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:04 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:05 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:05 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:06 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:06 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:06 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:07 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:08 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:08 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:09 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:09 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:09 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:10 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:10 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:11 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:12 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:12 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:12 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:13 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:13 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:13 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:15 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:15 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:15 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:16 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:16 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:16 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:17 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:18 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:18 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:19 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:19 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:19 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:20 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:21 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:21 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:22 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:22 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:22 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:23 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:23 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:24 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:25 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:25 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:25 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:26 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:26 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:27 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:28 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:28 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:28 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:29 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:29 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:30 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:31 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:31 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:31 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:32 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:33 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:33 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:34 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:34 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:34 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:35 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:36 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:36 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:37 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:37 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:37 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:38 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:38 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:39 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:40 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:40 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:40 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:41 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:41 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:42 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:43 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:43 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:43 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:44 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:44 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:45 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:46 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:46 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:46 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:47 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:47 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:48 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:49 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:49 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:49 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:50 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:50 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:50 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:51 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:51 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:52 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:53 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:53 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:53 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:54 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:54 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:55 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:56 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:56 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:56 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:57 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:57 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:57 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:59 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:59 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:39:59 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:00 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:00 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:00 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:02 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:02 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:02 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:03 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:03 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:03 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:04 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:05 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:05 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:06 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:06 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:06 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:07 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:08 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:08 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:09 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:09 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:09 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:10 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:11 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:11 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:12 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:12 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:12 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:13 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:13 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:14 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:15 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:15 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:15 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:16 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:16 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:17 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:18 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:18 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:18 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:19 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:19 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:20 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:21 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:21 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:21 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:22 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:22 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:22 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:23 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:24 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:24 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:25 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:25 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:25 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:26 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:27 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:27 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:28 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:28 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:28 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:29 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:30 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:30 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:31 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:31 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:31 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:32 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:33 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:33 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:34 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:34 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:34 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:35 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:35 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:36 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:37 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:37 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:37 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:38 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:38 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:39 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:40 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:40 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:40 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:41 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:41 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:42 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:43 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:43 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:43 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:44 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:44 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:45 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:46 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:46 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:46 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:47 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:47 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:48 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:49 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:49 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:49 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:50 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:50 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:51 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:52 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:52 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:52 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:53 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:53 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:53 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:54 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:55 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:55 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:56 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:56 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:56 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:57 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:58 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:58 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:59 [ 391 ] DEBUG : Executing query DETACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:59 [ 391 ] DEBUG : Executing query ATTACH TABLE test_mutations_with_ast_elements on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:59 [ 391 ] DEBUG : Executing query SELECT sum(is_done) FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 on test_mutations_with_merge_tree (cluster.py:3423, query) 2024-10-21 22:40:59 [ 391 ] DEBUG : Executing query SELECT mutation_id, command, parts_to_do, is_done FROM system.mutations WHERE table = 'test_mutations_with_ast_elements' SETTINGS force_index_by_date = 0, force_primary_key = 0 FORMAT TSVWithNames on test_mutations_with_merge_tree (cluster.py:3423, query) =============================== warnings summary =============================== test_keeper_mntr_pressure/test.py::test_aggressive_mntr /usr/local/lib/python3.10/dist-packages/_pytest/threadexception.py:73: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-141 () Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/ClickHouse/tests/integration/test_keeper_mntr_pressure/test.py", line 54, in node1_thread = threading.Thread(target=lambda: go_mntr(node1)) File "/ClickHouse/tests/integration/test_keeper_mntr_pressure/test.py", line 50, in go_mntr print(node.name, keeper_utils.send_4lw_cmd(cluster, node, "mntr")) File "/ClickHouse/tests/integration/helpers/keeper_utils.py", line 17, in send_4lw_cmd client = get_keeper_socket(cluster, node, port) File "/ClickHouse/tests/integration/helpers/keeper_utils.py", line 10, in get_keeper_socket client.connect((hosts, port)) ConnectionResetError: [Errno 104] Connection reset by peer warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 168.87s call test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task 53.79s call test_mask_sensitive_info/test.py::test_backup_to_s3 43.67s call test_keeper_nodes_move/test.py::test_node_move 38.52s call test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 37.24s call test_keeper_mntr_pressure/test.py::test_aggressive_mntr 31.69s setup test_replace_partition/test.py::test_normal_work 31.39s setup test_recovery_replica/test.py::test_choose_source_replica 31.36s call test_mask_sensitive_info/test.py::test_table_functions 28.36s setup test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch 28.12s call test_mask_sensitive_info/test.py::test_table_function_ways_to_call 25.25s call test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 25.13s call test_recovery_replica/test.py::test_update_metadata 23.67s setup test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 22.95s setup test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with 22.89s teardown test_log_family_s3/test.py::test_log_family_s3[TinyLog] 22.83s setup test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 22.58s teardown test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns 22.45s setup test_mask_sensitive_info/test.py::test_backup_to_s3 22.35s setup test_reloading_storage_configuration/test.py::test_add_disk 19.64s setup test_reload_zookeeper/test.py::test_reload_zookeeper 19.28s setup test_log_family_s3/test.py::test_log_family_s3[Log] 18.89s call test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 18.50s setup test_keeper_four_word_command/test.py::test_cmd_clrs 17.62s call test_reload_zookeeper/test.py::test_reload_zookeeper 17.61s call test_row_policy/test.py::test_change_of_users_xml_changes_row_policies 17.06s call test_mask_sensitive_info/test.py::test_create_table 16.65s setup test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 16.63s setup test_keeper_session/test.py::test_session_close_shutdown 16.48s setup test_merge_table_over_distributed/test.py::test_filtering 16.37s setup test_part_uuid/test.py::test_part_uuid 15.97s setup test_multi_access_storage_role_management/test.py::test_quotas 15.95s setup test_keeper_nodes_move/test.py::test_node_move 15.41s setup test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 15.41s setup test_rocksdb_options/test.py::test_invalid_column_family_options 15.30s setup test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 15.22s setup test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings 13.65s setup test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task 13.57s setup test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 12.26s call test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 11.68s call test_rocksdb_options/test.py::test_table_invalid_options 11.24s call test_rocksdb_options/test.py::test_table_invalid_column_family_options 11.12s setup test_keeper_four_word_command/test_allow_list.py::test_allow_list 10.94s call test_rocksdb_options/test.py::test_invalid_options 10.93s setup test_logs_level/test.py::test_check_client_logs_level 10.89s call test_rocksdb_options/test.py::test_invalid_column_family_options 10.65s call test_keeper_session/test.py::test_session_close_shutdown 10.57s call test_row_policy/test.py::test_dcl_introspection 10.52s call test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml 10.50s call test_row_policy/test.py::test_introspection 9.53s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 9.41s teardown test_reload_zookeeper/test.py::test_reload_zookeeper 8.69s call test_multi_access_storage_role_management/test.py::test_role_from_different_storages 7.76s call test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 7.50s teardown test_read_temporary_tables_on_failure/test.py::test_different_versions 7.47s call test_reloading_storage_configuration/test.py::test_remove_disk 7.35s call test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 7.28s call test_reloading_storage_configuration/test.py::test_add_disk_to_policy 7.25s call test_reloading_storage_configuration/test.py::test_remove_policy 7.24s call test_recovery_replica/test.py::test_choose_source_replica 7.23s call test_reloading_storage_configuration/test.py::test_new_policy_works 6.78s setup test_keeper_mntr_pressure/test.py::test_aggressive_mntr 6.75s call test_reloading_storage_configuration/test.py::test_add_policy 6.72s call test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 6.68s setup test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 6.60s call test_reloading_storage_configuration/test.py::test_add_disk 6.59s teardown test_recovery_replica/test.py::test_update_metadata 6.15s call test_reloading_storage_configuration/test.py::test_add_volume_to_policy 6.11s call test_reloading_storage_configuration/test.py::test_update_disk 6.01s teardown test_row_policy/test.py::test_smoke 5.87s call test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 5.60s teardown test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings 5.47s call test_recovery_replica/test.py::test_recovery 5.42s call test_read_temporary_tables_on_failure/test.py::test_different_versions 5.28s teardown test_replace_partition/test.py::test_normal_work 5.04s call test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with 5.04s call test_part_uuid/test.py::test_part_uuid 4.97s call test_multi_access_storage_role_management/test.py::test_settings_profiles 4.95s call test_multi_access_storage_role_management/test.py::test_users 4.92s teardown test_part_uuid/test.py::test_part_uuid 4.85s teardown test_mask_sensitive_info/test.py::test_table_functions 4.84s call test_multi_access_storage_role_management/test.py::test_quotas 4.78s setup test_read_temporary_tables_on_failure/test.py::test_different_versions 4.74s call test_multi_access_storage_role_management/test.py::test_roles 4.54s teardown test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed 4.40s call test_mask_sensitive_info/test.py::test_create_alter_user 4.27s teardown test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 4.20s teardown test_logs_level/test.py::test_check_client_logs_level 4.03s call test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 3.89s teardown test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 3.61s teardown test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop 3.42s teardown test_reloading_storage_configuration/test.py::test_update_disk 3.35s teardown test_replication_without_zookeeper/test.py::test_startup_without_zookeeper 3.22s call test_row_policy/test.py::test_join 2.82s call test_row_policy/test.py::test_policy_on_distributed_table_via_role 2.78s teardown test_keeper_nodes_move/test.py::test_node_move 2.77s call test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned 2.77s call test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 2.77s call test_row_policy/test.py::test_miscellaneous_engines 2.72s call test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 2.68s teardown test_keeper_mntr_pressure/test.py::test_aggressive_mntr 2.68s call test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 2.67s teardown test_keeper_session/test.py::test_session_timeout 2.65s call test_mask_sensitive_info/test.py::test_encryption_functions 2.65s call test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 2.62s call test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 2.53s teardown test_multi_access_storage_role_management/test.py::test_users 2.49s teardown test_keeper_four_word_command/test_allow_list.py::test_allow_list 2.47s teardown test_keeper_force_recovery_single_node/test.py::test_cluster_recovery 2.47s teardown test_keeper_four_word_command/test.py::test_cmd_wchs 2.47s call test_mask_sensitive_info/test.py::test_create_dictionary 2.42s call test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 2.33s teardown test_rocksdb_options/test.py::test_valid_options 2.31s call test_log_family_s3/test.py::test_log_family_s3[Log] 2.27s teardown test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with 2.22s call test_replace_partition/test.py::test_normal_work 2.15s call test_log_family_s3/test.py::test_log_family_s3[TinyLog] 2.14s call test_mask_sensitive_info/test.py::test_create_database 2.12s teardown test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 2.05s call test_log_family_s3/test.py::test_log_family_s3[StripeLog] 2.01s teardown test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery 1.94s call test_mask_sensitive_info/test.py::test_on_cluster 1.92s call test_row_policy/test.py::test_smoke 1.90s call test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch 1.44s call test_keeper_four_word_command/test_allow_list.py::test_allow_list 1.20s call test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns 1.17s call test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings 1.10s call test_rocksdb_options/test.py::test_valid_options 1.03s call test_keeper_four_word_command/test.py::test_cmd_rqld 0.83s call test_keeper_four_word_command/test.py::test_cmd_mntr 0.75s call test_merge_table_over_distributed/test.py::test_filtering 0.71s call test_keeper_four_word_command/test.py::test_cmd_lgif 0.65s call test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed 0.48s call test_merge_table_over_distributed/test.py::test_global_in 0.43s call test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster 0.37s call test_keeper_four_word_command/test.py::test_cmd_stat 0.34s call test_keeper_four_word_command/test.py::test_cmd_crst 0.31s call test_keeper_four_word_command/test.py::test_cmd_cons 0.29s call test_keeper_four_word_command/test.py::test_cmd_wchc 0.28s call test_keeper_four_word_command/test.py::test_cmd_dump 0.27s call test_keeper_four_word_command/test.py::test_cmd_wchp 0.27s call test_rocksdb_options/test.py::test_table_valid_options 0.27s call test_keeper_four_word_command/test.py::test_cmd_srvr 0.27s call test_keeper_four_word_command/test.py::test_cmd_clrs 0.23s call test_keeper_four_word_command/test.py::test_cmd_wchs 0.22s call test_rocksdb_options/test.py::test_valid_column_family_options 0.22s call test_rocksdb_options/test.py::test_table_valid_column_family_options 0.22s call test_logs_level/test.py::test_check_client_logs_level 0.17s call test_keeper_four_word_command/test.py::test_cmd_srst 0.13s call test_keeper_four_word_command/test.py::test_cmd_conf 0.08s call test_keeper_four_word_command/test.py::test_cmd_csnp 0.04s call test_keeper_session/test.py::test_session_timeout 0.03s teardown test_row_policy/test.py::test_grant_create_row_policy 0.02s teardown test_row_policy/test.py::test_dcl_introspection 0.02s call test_row_policy/test.py::test_dcl_management 0.02s call test_keeper_four_word_command/test.py::test_cmd_isro 0.02s call test_row_policy/test.py::test_grant_create_row_policy 0.02s call test_row_policy/test.py::test_row_policy_filter_with_subquery 0.02s teardown test_row_policy/test.py::test_introspection 0.02s teardown test_row_policy/test.py::test_change_of_users_xml_changes_row_policies 0.02s teardown test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml 0.02s teardown test_row_policy/test.py::test_row_policy_filter_with_subquery 0.02s teardown test_row_policy/test.py::test_dcl_management 0.02s teardown test_row_policy/test.py::test_policy_on_distributed_table_via_role 0.02s teardown test_row_policy/test.py::test_miscellaneous_engines 0.02s call test_row_policy/test.py::test_reload_users_xml_by_timer 0.02s teardown test_row_policy/test.py::test_reload_users_xml_by_timer 0.02s teardown test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned 0.02s teardown test_row_policy/test.py::test_join 0.01s call test_keeper_four_word_command/test.py::test_cmd_ruok 0.00s setup test_mask_sensitive_info/test.py::test_create_table 0.00s teardown test_recovery_replica/test.py::test_choose_source_replica 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk 0.00s teardown test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch 0.00s setup test_log_family_s3/test.py::test_log_family_s3[StripeLog] 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 0.00s setup test_reloading_storage_configuration/test.py::test_remove_disk 0.00s setup test_rocksdb_options/test.py::test_table_invalid_options 0.00s setup test_row_policy/test.py::test_introspection 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_mntr 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[Log] 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchp 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_lgif 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_stat 0.00s setup test_row_policy/test.py::test_dcl_management 0.00s setup test_row_policy/test.py::test_reload_users_xml_by_timer 0.00s teardown test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task 0.00s teardown test_mask_sensitive_info/test.py::test_create_database 0.00s setup test_row_policy/test.py::test_dcl_introspection 0.00s setup test_merge_table_over_distributed/test.py::test_global_in 0.00s teardown test_keeper_session/test.py::test_session_close_shutdown 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s setup test_row_policy/test.py::test_grant_create_row_policy 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_cons 0.00s setup test_row_policy/test.py::test_smoke 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] 0.00s setup test_reloading_storage_configuration/test.py::test_new_policy_works 0.00s setup test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchs 0.00s setup test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns 0.00s setup test_reloading_storage_configuration/test.py::test_remove_disk_from_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_remove_policy 0.00s teardown test_rocksdb_options/test.py::test_invalid_column_family_options 0.00s setup test_recovery_replica/test.py::test_update_metadata 0.00s setup test_rocksdb_options/test.py::test_valid_options 0.00s setup test_row_policy/test.py::test_row_policy_filter_with_subquery 0.00s teardown test_mask_sensitive_info/test.py::test_backup_to_s3 0.00s setup test_keeper_four_word_command/test.py::test_cmd_cons 0.00s setup test_row_policy/test.py::test_change_of_users_xml_changes_row_policies 0.00s teardown test_mask_sensitive_info/test.py::test_create_dictionary 0.00s teardown test_mask_sensitive_info/test.py::test_encryption_functions 0.00s setup test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_srvr 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_options 0.00s setup test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned 0.00s setup test_row_policy/test.py::test_join 0.00s setup test_log_family_s3/test.py::test_log_family_s3[TinyLog] 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchc 0.00s setup test_rocksdb_options/test.py::test_valid_column_family_options 0.00s setup test_row_policy/test.py::test_miscellaneous_engines 0.00s setup test_row_policy/test.py::test_policy_on_distributed_table_via_role 0.00s setup test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s setup test_mask_sensitive_info/test.py::test_create_dictionary 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 0.00s setup test_keeper_four_word_command/test.py::test_cmd_crst 0.00s setup test_rocksdb_options/test.py::test_table_valid_options 0.00s teardown test_rocksdb_options/test.py::test_table_valid_options 0.00s teardown test_rocksdb_options/test.py::test_table_valid_column_family_options 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] 0.00s teardown test_recovery_replica/test.py::test_recovery 0.00s setup test_reloading_storage_configuration/test.py::test_add_policy 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s setup test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_clrs 0.00s setup test_reloading_storage_configuration/test.py::test_remove_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_new_policy_works 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] 0.00s teardown test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_conf 0.00s setup test_multi_access_storage_role_management/test.py::test_users 0.00s setup test_keeper_four_word_command/test.py::test_cmd_mntr 0.00s teardown test_merge_table_over_distributed/test.py::test_global_in 0.00s teardown test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] 0.00s teardown test_reloading_storage_configuration/test.py::test_add_policy 0.00s teardown test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s setup test_mask_sensitive_info/test.py::test_on_cluster 0.00s teardown test_merge_table_over_distributed/test.py::test_filtering 0.00s teardown test_multi_access_storage_role_management/test.py::test_quotas 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_dump 0.00s setup test_keeper_four_word_command/test.py::test_cmd_rqld 0.00s setup test_keeper_four_word_command/test.py::test_cmd_dump 0.00s setup test_recovery_replica/test.py::test_recovery 0.00s setup test_reloading_storage_configuration/test.py::test_remove_volume_from_policy 0.00s setup test_reloading_storage_configuration/test.py::test_update_disk 0.00s setup test_mask_sensitive_info/test.py::test_table_functions 0.00s setup test_rocksdb_options/test.py::test_table_invalid_column_family_options 0.00s setup test_multi_access_storage_role_management/test.py::test_settings_profiles 0.00s setup test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed 0.00s teardown test_rocksdb_options/test.py::test_valid_column_family_options 0.00s teardown test_rocksdb_options/test.py::test_invalid_options 0.00s setup test_keeper_four_word_command/test.py::test_cmd_stat 0.00s setup test_reloading_storage_configuration/test.py::test_add_volume_to_policy 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_csnp 0.00s setup test_keeper_four_word_command/test.py::test_cmd_srvr 0.00s setup test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config 0.00s setup test_keeper_four_word_command/test.py::test_cmd_conf 0.00s teardown test_reloading_storage_configuration/test.py::test_add_disk_to_policy 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_crst 0.00s setup test_mask_sensitive_info/test.py::test_table_function_ways_to_call 0.00s teardown test_multi_access_storage_role_management/test.py::test_role_from_different_storages 0.00s setup test_multi_access_storage_role_management/test.py::test_roles 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_rqld 0.00s setup test_multi_access_storage_role_management/test.py::test_role_from_different_storages 0.00s teardown test_mask_sensitive_info/test.py::test_table_function_ways_to_call 0.00s setup test_keeper_four_word_command/test.py::test_cmd_isro 0.00s teardown test_multi_access_storage_role_management/test.py::test_roles 0.00s setup test_keeper_session/test.py::test_session_timeout 0.00s setup test_rocksdb_options/test.py::test_invalid_options 0.00s setup test_mask_sensitive_info/test.py::test_create_alter_user 0.00s teardown test_mask_sensitive_info/test.py::test_create_table 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_srst 0.00s teardown test_mask_sensitive_info/test.py::test_on_cluster 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_wchc 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_ruok 0.00s setup test_keeper_four_word_command/test.py::test_cmd_lgif 0.00s setup test_keeper_four_word_command/test.py::test_cmd_ruok 0.00s setup test_keeper_four_word_command/test.py::test_cmd_srst 0.00s setup test_keeper_four_word_command/test.py::test_cmd_csnp 0.00s setup test_keeper_four_word_command/test.py::test_cmd_wchp 0.00s setup test_mask_sensitive_info/test.py::test_encryption_functions 0.00s teardown test_multi_access_storage_role_management/test.py::test_settings_profiles 0.00s setup test_mask_sensitive_info/test.py::test_create_database 0.00s teardown test_log_family_s3/test.py::test_log_family_s3[StripeLog] 0.00s teardown test_mask_sensitive_info/test.py::test_create_alter_user 0.00s teardown test_keeper_four_word_command/test.py::test_cmd_isro =========================== short test summary info ============================ FAILED test_merge_table_over_distributed/test.py::test_global_in - helpers.cl... FAILED test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed FAILED test_row_policy/test.py::test_change_of_users_xml_changes_row_policies FAILED test_row_policy/test.py::test_dcl_introspection - helpers.client.Query... FAILED test_row_policy/test.py::test_dcl_management - Exception: Command ['do... FAILED test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml FAILED test_row_policy/test.py::test_grant_create_row_policy - Exception: Com... FAILED test_row_policy/test.py::test_introspection - helpers.client.QueryRunt... FAILED test_row_policy/test.py::test_join - helpers.client.QueryRuntimeExcept... FAILED test_row_policy/test.py::test_miscellaneous_engines - helpers.client.Q... FAILED test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned FAILED test_row_policy/test.py::test_policy_on_distributed_table_via_role - h... FAILED test_row_policy/test.py::test_reload_users_xml_by_timer - Exception: C... FAILED test_row_policy/test.py::test_row_policy_filter_with_subquery - Except... FAILED test_row_policy/test.py::test_smoke - helpers.client.QueryRuntimeExcep... FAILED test_mask_sensitive_info/test.py::test_encryption_functions - UnicodeD... FAILED test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster FAILED test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task ERROR test_row_policy/test.py::test_change_of_users_xml_changes_row_policies ERROR test_row_policy/test.py::test_dcl_introspection - Exception: Command ['... ERROR test_row_policy/test.py::test_dcl_management - Exception: Command ['doc... ERROR test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml - ... ERROR test_row_policy/test.py::test_grant_create_row_policy - Exception: Comm... ERROR test_row_policy/test.py::test_introspection - Exception: Command ['dock... ERROR test_row_policy/test.py::test_join - Exception: Command ['docker', 'exe... ERROR test_row_policy/test.py::test_miscellaneous_engines - Exception: Comman... ERROR test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned ERROR test_row_policy/test.py::test_policy_on_distributed_table_via_role - Ex... ERROR test_row_policy/test.py::test_reload_users_xml_by_timer - Exception: Co... ERROR test_row_policy/test.py::test_row_policy_filter_with_subquery - Excepti... ERROR test_row_policy/test.py::test_smoke - Exception: Command ['docker', 'ex... PASSED test_merge_table_over_distributed/test.py::test_filtering PASSED test_keeper_four_word_command/test.py::test_cmd_clrs PASSED test_keeper_four_word_command/test.py::test_cmd_conf PASSED test_keeper_four_word_command/test.py::test_cmd_cons PASSED test_keeper_four_word_command/test.py::test_cmd_crst PASSED test_keeper_four_word_command/test.py::test_cmd_csnp PASSED test_keeper_four_word_command/test.py::test_cmd_dump PASSED test_keeper_four_word_command/test.py::test_cmd_isro PASSED test_keeper_four_word_command/test.py::test_cmd_lgif PASSED test_multi_access_storage_role_management/test.py::test_quotas PASSED test_keeper_four_word_command/test.py::test_cmd_mntr PASSED test_log_family_s3/test.py::test_log_family_s3[Log] PASSED test_keeper_four_word_command/test.py::test_cmd_rqld PASSED test_keeper_four_word_command/test.py::test_cmd_ruok PASSED test_keeper_four_word_command/test.py::test_cmd_srst PASSED test_keeper_four_word_command/test.py::test_cmd_srvr PASSED test_keeper_four_word_command/test.py::test_cmd_stat PASSED test_keeper_four_word_command/test.py::test_cmd_wchc PASSED test_log_family_s3/test.py::test_log_family_s3[StripeLog] PASSED test_keeper_four_word_command/test.py::test_cmd_wchp PASSED test_keeper_four_word_command/test.py::test_cmd_wchs PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0] PASSED test_log_family_s3/test.py::test_log_family_s3[TinyLog] PASSED test_rocksdb_options/test.py::test_invalid_column_family_options PASSED test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1] PASSED test_reloading_storage_configuration/test.py::test_add_disk PASSED test_multi_access_storage_role_management/test.py::test_role_from_different_storages PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0] PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1] PASSED test_multi_access_storage_role_management/test.py::test_roles PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0] PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_policy PASSED test_rocksdb_options/test.py::test_invalid_options PASSED test_recovery_replica/test.py::test_choose_source_replica PASSED test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1] PASSED test_multi_access_storage_role_management/test.py::test_settings_profiles PASSED test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config PASSED test_recovery_replica/test.py::test_recovery PASSED test_multi_access_storage_role_management/test.py::test_users PASSED test_rocksdb_options/test.py::test_table_invalid_column_family_options PASSED test_reloading_storage_configuration/test.py::test_add_policy PASSED test_keeper_session/test.py::test_session_close_shutdown PASSED test_keeper_session/test.py::test_session_timeout PASSED test_reloading_storage_configuration/test.py::test_add_volume_to_policy PASSED test_rocksdb_options/test.py::test_table_invalid_options PASSED test_rocksdb_options/test.py::test_table_valid_column_family_options PASSED test_rocksdb_options/test.py::test_table_valid_options PASSED test_rocksdb_options/test.py::test_valid_column_family_options PASSED test_rocksdb_options/test.py::test_valid_options PASSED test_reloading_storage_configuration/test.py::test_new_policy_works PASSED test_recovery_replica/test.py::test_update_metadata PASSED test_reloading_storage_configuration/test.py::test_remove_disk PASSED test_mask_sensitive_info/test.py::test_backup_to_s3 PASSED test_keeper_force_recovery_single_node/test.py::test_cluster_recovery PASSED test_reloading_storage_configuration/test.py::test_remove_disk_from_policy PASSED test_mask_sensitive_info/test.py::test_create_alter_user PASSED test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings PASSED test_mask_sensitive_info/test.py::test_create_database PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch PASSED test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns PASSED test_mask_sensitive_info/test.py::test_create_dictionary PASSED test_reloading_storage_configuration/test.py::test_remove_policy PASSED test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit PASSED test_reloading_storage_configuration/test.py::test_remove_volume_from_policy PASSED test_keeper_four_word_command/test_allow_list.py::test_allow_list PASSED test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery PASSED test_reloading_storage_configuration/test.py::test_update_disk PASSED test_mask_sensitive_info/test.py::test_create_table PASSED test_mask_sensitive_info/test.py::test_on_cluster PASSED test_keeper_nodes_move/test.py::test_node_move PASSED test_part_uuid/test.py::test_part_uuid PASSED test_logs_level/test.py::test_check_client_logs_level PASSED test_replication_without_zookeeper/test.py::test_startup_without_zookeeper PASSED test_read_temporary_tables_on_failure/test.py::test_different_versions PASSED test_mask_sensitive_info/test.py::test_table_function_ways_to_call PASSED test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop PASSED test_keeper_mntr_pressure/test.py::test_aggressive_mntr PASSED test_replace_partition/test.py::test_normal_work PASSED test_mask_sensitive_info/test.py::test_table_functions PASSED test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table PASSED test_reload_zookeeper/test.py::test_reload_zookeeper ======= 18 failed, 82 passed, 1 warning, 13 errors in 302.09s (0:05:02) ======== Traceback (most recent call last): File "/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration/./runner", line 455, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_a8aptb --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/_temp/test/git-repo-copy/src/Server/grpc_protos:/ClickHouse/src/Server/grpc_protos --volume=/run:/run/host:ro --mount type=bind,source=/home/ubuntu/_work/_temp/test/dockerd_volume_dir,target=/var/lib/docker -e DOCKER_HELPER_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_BASE_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_KERBERIZED_HADOOP_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_KERBEROS_KDC_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_MYSQL_JAVA_CLIENT_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_MYSQL_JS_CLIENT_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_MYSQL_PHP_CLIENT_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_NGINX_DAV_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=0-46d4f9bd6a40e43bf5543be25cc704364e0c737b -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e CLICKHOUSE_USE_NEW_ANALYZER=1 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_keeper_force_recovery_single_node/test.py::test_cluster_recovery test_keeper_four_word_command/test.py::test_cmd_clrs test_keeper_four_word_command/test.py::test_cmd_conf test_keeper_four_word_command/test.py::test_cmd_cons test_keeper_four_word_command/test.py::test_cmd_crst test_keeper_four_word_command/test.py::test_cmd_csnp test_keeper_four_word_command/test.py::test_cmd_dump test_keeper_four_word_command/test.py::test_cmd_isro test_keeper_four_word_command/test.py::test_cmd_lgif test_keeper_four_word_command/test.py::test_cmd_mntr test_keeper_four_word_command/test.py::test_cmd_rqld test_keeper_four_word_command/test.py::test_cmd_ruok test_keeper_four_word_command/test.py::test_cmd_srst test_keeper_four_word_command/test.py::test_cmd_srvr test_keeper_four_word_command/test.py::test_cmd_stat test_keeper_four_word_command/test.py::test_cmd_wchc test_keeper_four_word_command/test.py::test_cmd_wchp test_keeper_four_word_command/test.py::test_cmd_wchs test_keeper_four_word_command/test_allow_list.py::test_allow_list test_keeper_mntr_pressure/test.py::test_aggressive_mntr test_keeper_nodes_move/test.py::test_node_move test_keeper_session/test.py::test_session_close_shutdown test_keeper_session/test.py::test_session_timeout test_keeper_snapshot_on_exit/test.py::test_snapshot_on_exit 'test_log_family_s3/test.py::test_log_family_s3[Log]' 'test_log_family_s3/test.py::test_log_family_s3[StripeLog]' 'test_log_family_s3/test.py::test_log_family_s3[TinyLog]' test_logs_level/test.py::test_check_client_logs_level test_mask_sensitive_info/test.py::test_backup_to_s3 test_mask_sensitive_info/test.py::test_create_alter_user test_mask_sensitive_info/test.py::test_create_database test_mask_sensitive_info/test.py::test_create_dictionary test_mask_sensitive_info/test.py::test_create_table test_mask_sensitive_info/test.py::test_encryption_functions test_mask_sensitive_info/test.py::test_on_cluster test_mask_sensitive_info/test.py::test_table_function_ways_to_call test_mask_sensitive_info/test.py::test_table_functions test_materialized_view_restart_server/test.py::test_materialized_view_with_subquery test_merge_table_over_distributed/test.py::test_filtering test_merge_table_over_distributed/test.py::test_global_in test_merge_table_over_distributed/test.py::test_select_table_name_from_merge_over_distributed test_multi_access_storage_role_management/test.py::test_quotas test_multi_access_storage_role_management/test.py::test_role_from_different_storages test_multi_access_storage_role_management/test.py::test_roles test_multi_access_storage_role_management/test.py::test_settings_profiles test_multi_access_storage_role_management/test.py::test_users test_mutations_with_merge_tree/test.py::test_mutations_with_merge_background_task test_mutations_with_merge_tree/test.py::test_mutations_with_truncate_table 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_error_on_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_no_unavailable_shards[1]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[0]' 'test_parallel_replicas_distributed_skip_shards/test.py::test_skip_unavailable_shards[1]' test_part_uuid/test.py::test_part_uuid test_passing_max_partitions_to_read_remotely/test.py::test_default_database_on_cluster test_read_temporary_tables_on_failure/test.py::test_different_versions test_recovery_replica/test.py::test_choose_source_replica test_recovery_replica/test.py::test_recovery test_recovery_replica/test.py::test_update_metadata test_reload_max_table_size_to_drop/test.py::test_reload_max_table_size_to_drop test_reload_zookeeper/test.py::test_reload_zookeeper test_reloading_storage_configuration/test.py::test_add_disk test_reloading_storage_configuration/test.py::test_add_disk_to_policy test_reloading_storage_configuration/test.py::test_add_disk_to_separate_config test_reloading_storage_configuration/test.py::test_add_policy test_reloading_storage_configuration/test.py::test_add_volume_to_policy test_reloading_storage_configuration/test.py::test_new_policy_works test_reloading_storage_configuration/test.py::test_remove_disk test_reloading_storage_configuration/test.py::test_remove_disk_from_policy test_reloading_storage_configuration/test.py::test_remove_policy test_reloading_storage_configuration/test.py::test_remove_volume_from_policy test_reloading_storage_configuration/test.py::test_update_disk test_replace_partition/test.py::test_normal_work test_replicated_merge_tree_config/test.py::test_replicated_merge_tree_settings test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_after_fetch test_replicated_s3_zero_copy_drop_partition/test.py::test_drop_complex_columns test_replication_without_zookeeper/test.py::test_startup_without_zookeeper test_rocksdb_options/test.py::test_invalid_column_family_options test_rocksdb_options/test.py::test_invalid_options test_rocksdb_options/test.py::test_table_invalid_column_family_options test_rocksdb_options/test.py::test_table_invalid_options test_rocksdb_options/test.py::test_table_valid_column_family_options test_rocksdb_options/test.py::test_table_valid_options test_rocksdb_options/test.py::test_valid_column_family_options test_rocksdb_options/test.py::test_valid_options test_row_policy/test.py::test_cannot_trick_row_policy_with_keyword_with test_row_policy/test.py::test_change_of_users_xml_changes_row_policies test_row_policy/test.py::test_dcl_introspection test_row_policy/test.py::test_dcl_management test_row_policy/test.py::test_dcl_users_with_policies_from_users_xml test_row_policy/test.py::test_grant_create_row_policy test_row_policy/test.py::test_introspection test_row_policy/test.py::test_join test_row_policy/test.py::test_miscellaneous_engines test_row_policy/test.py::test_policy_from_users_xml_affects_only_user_assigned test_row_policy/test.py::test_policy_on_distributed_table_via_role test_row_policy/test.py::test_reload_users_xml_by_timer test_row_policy/test.py::test_row_policy_filter_with_subquery test_row_policy/test.py::test_smoke -vvv" altinityinfra/integration-tests-runner:0-46d4f9bd6a40e43bf5543be25cc704364e0c737b ' returned non-zero exit status 1.